2024-03-04T15:47:36.580 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-03-04T15:47:36.580 DEBUG:teuthology.run:Teuthology command: teuthology --name tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi --verbose --description orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} --owner scheduled_tonay@teuthology --archive /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581240 -- /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581240/orig.config.yaml 2024-03-04T15:47:36.610 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T15:47:36.701 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581240 branch: wip-asm0deuz-testing-2024-03-04-1211 description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} email: tonay@ibm.com first_in_suite: false job_id: '7581240' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-asm0deuz-testing-2024-03-04-1211 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 workunit: branch: wip-asm0deuz-testing-2024-03-04-1211 sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 owner: scheduled_tonay@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 3905 sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 sleep_before_teardown: 0 subset: 1/14 suite: orch:cephadm suite_branch: wip-asm0deuz-testing-2024-03-04-1211 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 targets: smithi083.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBEMT0OhSFWXWUnXcQSt30R69b09/MyThuIpLwcGR3YBD9YdlReB110ouWA539I2tqCPsShVn+qCZP3vdyTOSQs= smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD8xljKq9p545rXF3lnZ87pN5AxQ2dPNhjW/ghJCiH9Evyz3TTVYN5XcpAI3B7Ck+Kk0IOYt9e+LqpK3ksCGAUQ= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-03-04_15:07:38 tube: smithi user: tonay verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-04T15:47:36.701 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa; will attempt to use it 2024-03-04T15:47:36.702 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa/tasks 2024-03-04T15:47:36.702 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-04T15:47:36.704 INFO:teuthology.task.internal:Checking packages... 2024-03-04T15:47:36.736 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9ca0b5de35ed230e84387abdd8e619ed8e4f3282' 2024-03-04T15:47:36.736 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-04T15:47:36.736 INFO:teuthology.packaging:ref: None 2024-03-04T15:47:36.736 INFO:teuthology.packaging:tag: None 2024-03-04T15:47:36.736 INFO:teuthology.packaging:branch: wip-asm0deuz-testing-2024-03-04-1211 2024-03-04T15:47:36.736 INFO:teuthology.packaging:sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:47:36.737 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-asm0deuz-testing-2024-03-04-1211 2024-03-04T15:47:36.967 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1826.g9ca0b5de 2024-03-04T15:47:36.969 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-04T15:47:36.977 INFO:teuthology.task.internal:no buildpackages task found 2024-03-04T15:47:36.977 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-04T15:47:36.986 INFO:teuthology.task.internal:Saving configuration 2024-03-04T15:47:36.999 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-04T15:47:37.005 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T15:47:37.045 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581240', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:42:33.936299', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBEMT0OhSFWXWUnXcQSt30R69b09/MyThuIpLwcGR3YBD9YdlReB110ouWA539I2tqCPsShVn+qCZP3vdyTOSQs='} 2024-03-04T15:47:37.075 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581240', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:42:33.938259', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD8xljKq9p545rXF3lnZ87pN5AxQ2dPNhjW/ghJCiH9Evyz3TTVYN5XcpAI3B7Ck+Kk0IOYt9e+LqpK3ksCGAUQ='} 2024-03-04T15:47:37.075 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-04T15:47:37.081 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-03-04T15:47:37.082 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-04T15:47:37.082 INFO:teuthology.run_tasks:Running task console_log... 2024-03-04T15:47:37.174 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fae979820d0>, signals=[15]) 2024-03-04T15:47:37.174 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-04T15:47:37.181 INFO:teuthology.task.internal:Opening connections... 2024-03-04T15:47:37.181 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi083.front.sepia.ceph.com 2024-03-04T15:47:37.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:47:37.295 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2024-03-04T15:47:37.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:47:37.391 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-04T15:47:37.399 DEBUG:teuthology.orchestra.run.smithi083:> uname -m 2024-03-04T15:47:37.417 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64 2024-03-04T15:47:37.417 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release 2024-03-04T15:47:37.474 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="CentOS Stream" 2024-03-04T15:47:37.475 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="9" 2024-03-04T15:47:37.475 INFO:teuthology.orchestra.run.smithi083.stdout:ID="centos" 2024-03-04T15:47:37.475 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE="rhel fedora" 2024-03-04T15:47:37.475 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="9" 2024-03-04T15:47:37.475 INFO:teuthology.orchestra.run.smithi083.stdout:PLATFORM_ID="platform:el9" 2024-03-04T15:47:37.475 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T15:47:37.475 INFO:teuthology.orchestra.run.smithi083.stdout:ANSI_COLOR="0;31" 2024-03-04T15:47:37.475 INFO:teuthology.orchestra.run.smithi083.stdout:LOGO="fedora-logo-icon" 2024-03-04T15:47:37.475 INFO:teuthology.orchestra.run.smithi083.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T15:47:37.475 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://centos.org/" 2024-03-04T15:47:37.475 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T15:47:37.475 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T15:47:37.475 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T15:47:37.476 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2024-03-04T15:47:37.513 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2024-03-04T15:47:37.530 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2024-03-04T15:47:37.530 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2024-03-04T15:47:37.590 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2024-03-04T15:47:37.591 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="9" 2024-03-04T15:47:37.591 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2024-03-04T15:47:37.591 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2024-03-04T15:47:37.591 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="9" 2024-03-04T15:47:37.591 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el9" 2024-03-04T15:47:37.591 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T15:47:37.591 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2024-03-04T15:47:37.591 INFO:teuthology.orchestra.run.smithi086.stdout:LOGO="fedora-logo-icon" 2024-03-04T15:47:37.591 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T15:47:37.591 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2024-03-04T15:47:37.591 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T15:47:37.591 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T15:47:37.592 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T15:47:37.592 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2024-03-04T15:47:37.621 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-04T15:47:37.630 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-04T15:47:37.636 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-04T15:47:37.636 DEBUG:teuthology.orchestra.run.smithi083:> test '!' -e /home/ubuntu/cephtest 2024-03-04T15:47:37.639 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2024-03-04T15:47:37.654 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-04T15:47:37.661 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-04T15:47:37.662 DEBUG:teuthology.orchestra.run.smithi083:> test -z $(ls -A /var/lib/ceph) 2024-03-04T15:47:37.696 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2024-03-04T15:47:37.748 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-04T15:47:37.817 INFO:teuthology.run_tasks:Running task kernel... 2024-03-04T15:47:37.857 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-04T15:47:37.858 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-04T15:47:37.858 DEBUG:teuthology.orchestra.run.smithi083:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T15:47:37.858 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T15:47:37.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:47:37.875 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-03-04T15:47:37.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:47:37.876 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-03-04T15:47:37.932 INFO:teuthology.orchestra.run.smithi083.stdout:5.14.0-361.el9.x86_64 2024-03-04T15:47:37.932 INFO:teuthology.task.kernel:Running kernel on smithi083: 5.14.0-361.el9.x86_64 2024-03-04T15:47:37.932 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum install -y kernel 2024-03-04T15:47:37.933 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-361.el9.x86_64 2024-03-04T15:47:37.933 INFO:teuthology.task.kernel:Running kernel on smithi086: 5.14.0-361.el9.x86_64 2024-03-04T15:47:37.933 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2024-03-04T15:47:40.048 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-04T15:47:40.235 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-04T15:47:41.523 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-04T15:47:41.660 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-04T15:47:42.374 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-04T15:47:42.436 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-04T15:47:47.248 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 5.0 MB/s | 19 MB 00:03 2024-03-04T15:47:51.184 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-04T15:47:53.799 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 10 kB/s | 15 kB 00:01 2024-03-04T15:47:57.142 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 6.4 MB/s | 20 MB 00:03 2024-03-04T15:48:02.008 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-04T15:48:04.299 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-04T15:48:04.299 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-04T15:48:04.397 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-04T15:48:04.403 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-04T15:48:04.403 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-03-04T15:48:04.403 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-04T15:48:04.403 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-04T15:48:04.403 INFO:teuthology.orchestra.run.smithi086.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-04T15:48:04.403 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-04T15:48:04.403 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-04T15:48:04.403 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-04T15:48:04.403 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-04T15:48:04.404 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T15:48:04.404 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-04T15:48:04.404 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-04T15:48:04.404 INFO:teuthology.orchestra.run.smithi086.stdout:Install 4 Packages 2024-03-04T15:48:04.404 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T15:48:04.404 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 100 M 2024-03-04T15:48:04.405 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 123 M 2024-03-04T15:48:04.405 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-04T15:48:04.893 INFO:teuthology.orchestra.run.smithi086.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 21 MB/s | 6.0 MB 00:00 2024-03-04T15:48:05.177 INFO:teuthology.orchestra.run.smithi086.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 37 MB/s | 21 MB 00:00 2024-03-04T15:48:05.561 INFO:teuthology.orchestra.run.smithi086.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 50 MB/s | 33 MB 00:00 2024-03-04T15:48:05.725 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - AppStream 885 kB/s | 19 MB 00:22 2024-03-04T15:48:05.964 INFO:teuthology.orchestra.run.smithi086.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 29 MB/s | 39 MB 00:01 2024-03-04T15:48:05.965 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:48:05.965 INFO:teuthology.orchestra.run.smithi086.stdout:Total 64 MB/s | 100 MB 00:01 2024-03-04T15:48:06.851 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-04T15:48:07.205 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-04T15:48:07.206 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-04T15:48:08.570 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-04T15:48:08.571 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-04T15:48:10.003 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.1 MB 00:00 2024-03-04T15:48:11.547 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-04T15:48:11.585 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2024-03-04T15:48:12.211 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:48:12.344 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:48:13.651 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2024-03-04T15:48:15.119 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:48:15.312 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:48:18.578 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-04T15:48:20.087 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:48:20.264 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:48:20.760 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-04T15:48:20.760 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-04T15:48:20.842 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-04T15:48:20.852 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-04T15:48:20.852 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-03-04T15:48:20.852 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-04T15:48:20.852 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-04T15:48:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-04T15:48:20.853 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-04T15:48:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-04T15:48:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-04T15:48:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-04T15:48:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-04T15:48:20.853 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-04T15:48:20.853 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-04T15:48:20.853 INFO:teuthology.orchestra.run.smithi083.stdout:Install 4 Packages 2024-03-04T15:48:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-04T15:48:20.854 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 100 M 2024-03-04T15:48:20.854 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 123 M 2024-03-04T15:48:20.854 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-04T15:48:21.179 INFO:teuthology.orchestra.run.smithi083.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 23 MB/s | 6.0 MB 00:00 2024-03-04T15:48:21.446 INFO:teuthology.orchestra.run.smithi083.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 39 MB/s | 21 MB 00:00 2024-03-04T15:48:21.896 INFO:teuthology.orchestra.run.smithi083.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 40 MB/s | 39 MB 00:00 2024-03-04T15:48:22.213 INFO:teuthology.orchestra.run.smithi083.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 32 MB/s | 33 MB 00:01 2024-03-04T15:48:22.216 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:48:22.217 INFO:teuthology.orchestra.run.smithi083.stdout:Total 73 MB/s | 100 MB 00:01 2024-03-04T15:48:23.108 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-04T15:48:23.483 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-04T15:48:23.483 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-04T15:48:25.021 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:48:25.159 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-04T15:48:25.159 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-04T15:48:28.114 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-04T15:48:28.774 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:48:28.990 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:48:31.777 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:48:31.961 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:48:36.681 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:48:36.888 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:48:41.648 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:48:54.385 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:48:54.386 INFO:teuthology.orchestra.run.smithi086.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T15:48:54.386 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T15:48:54.470 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:48:54.892 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:48:54.893 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:48:54.893 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:48:54.893 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:48:55.323 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:48:55.323 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T15:48:55.323 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-04T15:48:55.323 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-04T15:48:55.323 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-04T15:48:55.323 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-04T15:48:55.323 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-04T15:48:55.323 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T15:48:55.323 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-04T15:48:55.820 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:48:55.967 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:48:55.968 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:48:55.968 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-04T15:48:55.968 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T15:48:55.968 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T15:48:55.969 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-04T15:48:55.969 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi086.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T15:48:55.969 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:48:56.088 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:48:56.088 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:48:56.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo rpm -qi grub2-tools 2024-03-04T15:48:56.122 INFO:teuthology.orchestra.run.smithi086.stdout:Name : grub2-tools 2024-03-04T15:48:56.122 INFO:teuthology.orchestra.run.smithi086.stdout:Epoch : 1 2024-03-04T15:48:56.122 INFO:teuthology.orchestra.run.smithi086.stdout:Version : 2.06 2024-03-04T15:48:56.122 INFO:teuthology.orchestra.run.smithi086.stdout:Release : 68.el9 2024-03-04T15:48:56.122 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: x86_64 2024-03-04T15:48:56.122 INFO:teuthology.orchestra.run.smithi086.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T15:48:56.122 INFO:teuthology.orchestra.run.smithi086.stdout:Group : Unspecified 2024-03-04T15:48:56.122 INFO:teuthology.orchestra.run.smithi086.stdout:Size : 8264254 2024-03-04T15:48:56.122 INFO:teuthology.orchestra.run.smithi086.stdout:License : GPLv3+ 2024-03-04T15:48:56.123 INFO:teuthology.orchestra.run.smithi086.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T15:48:56.123 INFO:teuthology.orchestra.run.smithi086.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T15:48:56.123 INFO:teuthology.orchestra.run.smithi086.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T15:48:56.123 INFO:teuthology.orchestra.run.smithi086.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T15:48:56.123 INFO:teuthology.orchestra.run.smithi086.stdout:Packager : builder@centos.org 2024-03-04T15:48:56.123 INFO:teuthology.orchestra.run.smithi086.stdout:Vendor : CentOS 2024-03-04T15:48:56.123 INFO:teuthology.orchestra.run.smithi086.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T15:48:56.123 INFO:teuthology.orchestra.run.smithi086.stdout:Summary : Support tools for GRUB. 2024-03-04T15:48:56.123 INFO:teuthology.orchestra.run.smithi086.stdout:Description : 2024-03-04T15:48:56.123 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T15:48:56.123 INFO:teuthology.orchestra.run.smithi086.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T15:48:56.123 INFO:teuthology.orchestra.run.smithi086.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T15:48:56.123 INFO:teuthology.orchestra.run.smithi086.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T15:48:56.123 INFO:teuthology.orchestra.run.smithi086.stdout:hardware devices. 2024-03-04T15:48:56.124 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T15:48:56.124 INFO:teuthology.orchestra.run.smithi086.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T15:48:56.124 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T15:48:56.125 INFO:teuthology.task.kernel:Updating grub on smithi086 to boot 5.14.0-427.el9.x86_64 2024-03-04T15:48:56.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T15:48:56.711 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2024-03-04T15:48:58.147 INFO:teuthology.orchestra.run.smithi086.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T15:48:58.176 INFO:teuthology.orchestra.run.smithi086.stderr:done 2024-03-04T15:48:58.179 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-03-04T15:48:58.194 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.hHGYhPvcay 2024-03-04T15:48:58.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.hHGYhPvcay 2024-03-04T15:48:58.262 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.hHGYhPvcay 2024-03-04T15:48:58.403 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.hHGYhPvcay is 6KB 2024-03-04T15:48:58.451 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.hHGYhPvcay 2024-03-04T15:48:58.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T15:48:58.531 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T15:48:58.531 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T15:48:58.531 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T15:48:58.531 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-04T15:48:58.532 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-04T15:48:58.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo shutdown -r now 2024-03-04T15:49:10.924 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:49:10.925 INFO:teuthology.orchestra.run.smithi083.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T15:49:10.925 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-04T15:49:11.009 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:49:11.572 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:49:11.572 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:49:11.572 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:49:11.572 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:49:11.995 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:49:11.996 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-04T15:49:11.996 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-04T15:49:11.996 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-04T15:49:11.996 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-04T15:49:11.996 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-04T15:49:11.996 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-04T15:49:11.996 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-04T15:49:11.996 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-04T15:49:12.537 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:49:12.697 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:49:12.697 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:49:12.697 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-04T15:49:12.697 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T15:49:12.697 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T15:49:12.697 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-04T15:49:12.697 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi083.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T15:49:12.698 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:49:12.818 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:49:12.818 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:49:12.818 DEBUG:teuthology.orchestra.run.smithi083:> sudo rpm -qi grub2-tools 2024-03-04T15:49:12.853 INFO:teuthology.orchestra.run.smithi083.stdout:Name : grub2-tools 2024-03-04T15:49:12.853 INFO:teuthology.orchestra.run.smithi083.stdout:Epoch : 1 2024-03-04T15:49:12.853 INFO:teuthology.orchestra.run.smithi083.stdout:Version : 2.06 2024-03-04T15:49:12.853 INFO:teuthology.orchestra.run.smithi083.stdout:Release : 68.el9 2024-03-04T15:49:12.853 INFO:teuthology.orchestra.run.smithi083.stdout:Architecture: x86_64 2024-03-04T15:49:12.853 INFO:teuthology.orchestra.run.smithi083.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T15:49:12.853 INFO:teuthology.orchestra.run.smithi083.stdout:Group : Unspecified 2024-03-04T15:49:12.854 INFO:teuthology.orchestra.run.smithi083.stdout:Size : 8264254 2024-03-04T15:49:12.854 INFO:teuthology.orchestra.run.smithi083.stdout:License : GPLv3+ 2024-03-04T15:49:12.854 INFO:teuthology.orchestra.run.smithi083.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T15:49:12.854 INFO:teuthology.orchestra.run.smithi083.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T15:49:12.854 INFO:teuthology.orchestra.run.smithi083.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T15:49:12.854 INFO:teuthology.orchestra.run.smithi083.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T15:49:12.854 INFO:teuthology.orchestra.run.smithi083.stdout:Packager : builder@centos.org 2024-03-04T15:49:12.854 INFO:teuthology.orchestra.run.smithi083.stdout:Vendor : CentOS 2024-03-04T15:49:12.854 INFO:teuthology.orchestra.run.smithi083.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T15:49:12.854 INFO:teuthology.orchestra.run.smithi083.stdout:Summary : Support tools for GRUB. 2024-03-04T15:49:12.854 INFO:teuthology.orchestra.run.smithi083.stdout:Description : 2024-03-04T15:49:12.854 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-04T15:49:12.854 INFO:teuthology.orchestra.run.smithi083.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T15:49:12.854 INFO:teuthology.orchestra.run.smithi083.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T15:49:12.854 INFO:teuthology.orchestra.run.smithi083.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T15:49:12.855 INFO:teuthology.orchestra.run.smithi083.stdout:hardware devices. 2024-03-04T15:49:12.855 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-04T15:49:12.855 INFO:teuthology.orchestra.run.smithi083.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T15:49:12.856 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T15:49:12.856 INFO:teuthology.task.kernel:Updating grub on smithi083 to boot 5.14.0-427.el9.x86_64 2024-03-04T15:49:12.856 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T15:49:13.456 INFO:teuthology.orchestra.run.smithi083.stderr:Generating grub configuration file ... 2024-03-04T15:49:14.911 INFO:teuthology.orchestra.run.smithi083.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T15:49:14.946 INFO:teuthology.orchestra.run.smithi083.stderr:done 2024-03-04T15:49:14.948 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-03-04T15:49:14.963 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.m8rOELNJ8e 2024-03-04T15:49:14.963 DEBUG:teuthology.orchestra.run.smithi083:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.m8rOELNJ8e 2024-03-04T15:49:15.032 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.m8rOELNJ8e 2024-03-04T15:49:15.171 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.m8rOELNJ8e is 6KB 2024-03-04T15:49:15.220 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.m8rOELNJ8e 2024-03-04T15:49:15.234 DEBUG:teuthology.orchestra.run.smithi083:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T15:49:15.298 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T15:49:15.299 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T15:49:15.299 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T15:49:15.299 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-04T15:49:15.300 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-04T15:49:15.452 DEBUG:teuthology.orchestra.run.smithi083:> sudo shutdown -r now 2024-03-04T15:49:28.687 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-04T15:49:28.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-04T15:49:28.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:49:45.472 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-04T15:49:45.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-04T15:49:45.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:49:47.196 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-03-04T15:49:56.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-04T15:49:56.205 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:49:59.296 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-03-04T15:50:03.836 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-03-04T15:50:11.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-04T15:50:11.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:50:12.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-04T15:50:12.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:50:15.932 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-03-04T15:50:27.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-04T15:50:27.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:50:43.113 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-03-04T15:50:43.598 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-04T15:50:43.598 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-04T15:50:43.598 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-03-04T15:50:43.612 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-427.el9.x86_64 2024-03-04T15:50:43.612 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-04T15:50:43.612 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T15:50:43.612 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T15:50:44.613 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-04T15:50:44.613 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-04T15:50:44.732 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2024-03-04T15:50:44.779 DEBUG:teuthology.parallel:result is None 2024-03-04T15:50:46.336 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-03-04T15:51:01.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-04T15:51:01.338 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:51:01.665 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-03-04T15:51:02.043 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-04T15:51:02.043 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-04T15:51:02.044 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-03-04T15:51:02.057 INFO:teuthology.orchestra.run.smithi083.stdout:5.14.0-427.el9.x86_64 2024-03-04T15:51:02.058 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-04T15:51:02.058 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T15:51:02.058 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T15:51:03.059 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-04T15:51:03.059 DEBUG:teuthology.orchestra.run.smithi083:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-04T15:51:03.203 INFO:teuthology.orchestra.run.smithi083.stdout:ttyS1 2024-03-04T15:51:03.235 DEBUG:teuthology.parallel:result is None 2024-03-04T15:51:03.236 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-04T15:51:03.243 INFO:teuthology.task.internal:Creating test directory... 2024-03-04T15:51:03.243 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T15:51:03.246 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T15:51:03.269 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-04T15:51:03.276 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-04T15:51:03.284 INFO:teuthology.task.internal:Creating archive directory... 2024-03-04T15:51:03.284 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T15:51:03.311 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T15:51:03.395 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-04T15:51:03.402 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-04T15:51:03.403 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T15:51:03.405 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T15:51:03.445 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:51:03.459 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:51:03.584 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:51:03.605 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:51:03.607 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-04T15:51:03.614 INFO:teuthology.task.internal:Configuring sudo... 2024-03-04T15:51:03.615 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T15:51:03.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T15:51:03.663 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-04T15:51:03.675 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-04T15:51:03.675 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T15:51:03.677 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T15:51:03.705 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:51:03.803 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:51:03.861 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:51:03.919 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:51:03.971 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-04T15:51:03.971 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T15:51:04.061 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:51:04.115 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:51:04.166 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:51:04.217 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:51:04.253 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-04T15:51:04.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T15:51:04.322 DEBUG:teuthology.orchestra.run.smithi083:> sudo service rsyslog restart 2024-03-04T15:51:04.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2024-03-04T15:51:04.370 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T15:51:04.429 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T15:51:04.664 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-04T15:51:04.672 INFO:teuthology.task.internal:Starting timer... 2024-03-04T15:51:04.672 INFO:teuthology.run_tasks:Running task pcp... 2024-03-04T15:51:04.684 INFO:teuthology.run_tasks:Running task selinux... 2024-03-04T15:51:04.692 DEBUG:teuthology.orchestra.run.smithi083:> sudo service auditd rotate 2024-03-04T15:51:04.771 INFO:teuthology.orchestra.run.smithi083.stdout:Rotating logs: 2024-03-04T15:51:04.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2024-03-04T15:51:04.841 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2024-03-04T15:51:04.843 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-04T15:51:04.843 DEBUG:teuthology.orchestra.run.smithi083:> /usr/sbin/getenforce 2024-03-04T15:51:04.862 INFO:teuthology.orchestra.run.smithi083.stdout:Permissive 2024-03-04T15:51:04.862 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2024-03-04T15:51:04.890 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2024-03-04T15:51:04.890 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi083.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi086.front.sepia.ceph.com': 'permissive'} 2024-03-04T15:51:04.890 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T15:51:04.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:51:04.929 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T15:51:04.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:51:04.958 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-04T15:51:04.958 DEBUG:teuthology.orchestra.run.smithi083:> sudo /usr/sbin/setenforce permissive 2024-03-04T15:51:05.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2024-03-04T15:51:05.042 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-04T15:51:05.052 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-04T15:51:05.057 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-04T15:51:05.273 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-04T15:51:05.290 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-04T15:51:05.291 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi083.front.sepia.ceph.com,smithi086.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-04T15:55:17.268 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi083.front.sepia.ceph.com'), Remote(name='ubuntu@smithi086.front.sepia.ceph.com')] 2024-03-04T15:55:17.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-04T15:55:17.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:55:17.368 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-03-04T15:55:17.446 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-04T15:55:17.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-04T15:55:17.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:55:17.562 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-03-04T15:55:17.648 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-04T15:55:17.649 INFO:teuthology.run_tasks:Running task clock... 2024-03-04T15:55:17.659 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-04T15:55:17.660 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T15:55:17.660 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T15:55:17.663 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T15:55:17.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T15:55:17.692 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T15:55:17.708 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T15:55:17.722 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T15:55:17.737 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: ntpd: command not found 2024-03-04T15:55:17.739 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T15:55:17.751 INFO:teuthology.orchestra.run.smithi083.stdout:506 Cannot talk to daemon 2024-03-04T15:55:17.767 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T15:55:17.770 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2024-03-04T15:55:17.784 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2024-03-04T15:55:17.784 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T15:55:17.801 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T15:55:17.817 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T15:55:17.838 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-03-04T15:55:17.841 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T15:55:17.841 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-03-04T15:55:17.841 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:55:17.841 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:55:17.841 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:55:17.841 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:55:17.872 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-03-04T15:55:17.875 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T15:55:17.875 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-03-04T15:55:17.875 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:55:17.875 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:55:17.875 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:55:17.875 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:55:17.876 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-04T15:55:17.974 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282'} 2024-03-04T15:55:17.974 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-03-04T15:55:17.975 INFO:tasks.cephadm:Cluster fsid is 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:55:17.975 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-04T15:55:17.975 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.83', 'mon.c': '[v2:172.21.15.83:3301,v1:172.21.15.83:6790]', 'mon.b': '172.21.15.86'} 2024-03-04T15:55:17.975 INFO:tasks.cephadm:First mon is mon.a on smithi083 2024-03-04T15:55:17.975 INFO:tasks.cephadm:First mgr is y 2024-03-04T15:55:17.975 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-04T15:55:17.976 DEBUG:teuthology.orchestra.run.smithi083:> sudo hostname $(hostname -s) 2024-03-04T15:55:18.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo hostname $(hostname -s) 2024-03-04T15:55:18.032 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-03-04T15:55:18.033 DEBUG:teuthology.orchestra.run.smithi083:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-04T15:55:18.321 INFO:teuthology.orchestra.run.smithi083.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 4 15:55 /home/ubuntu/cephtest/cephadm 2024-03-04T15:55:18.321 DEBUG:teuthology.orchestra.run.smithi086:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-04T15:55:18.577 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 4 15:55 /home/ubuntu/cephtest/cephadm 2024-03-04T15:55:18.578 DEBUG:teuthology.orchestra.run.smithi083:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-04T15:55:18.593 DEBUG:teuthology.orchestra.run.smithi086:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-04T15:55:18.619 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-03-04T15:55:18.620 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-04T15:55:18.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-04T15:55:19.481 INFO:teuthology.orchestra.run.smithi083.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-04T15:55:19.530 INFO:teuthology.orchestra.run.smithi086.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-04T15:55:41.706 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-04T15:55:41.707 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-04T15:55:41.707 INFO:teuthology.orchestra.run.smithi086.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-04T15:55:41.707 INFO:teuthology.orchestra.run.smithi086.stdout: "repo_digests": [ 2024-03-04T15:55:41.707 INFO:teuthology.orchestra.run.smithi086.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-04T15:55:41.707 INFO:teuthology.orchestra.run.smithi086.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-04T15:55:41.707 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-04T15:55:41.707 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-04T15:55:46.346 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T15:55:46.346 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-04T15:55:46.347 INFO:teuthology.orchestra.run.smithi083.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-04T15:55:46.347 INFO:teuthology.orchestra.run.smithi083.stdout: "repo_digests": [ 2024-03-04T15:55:46.347 INFO:teuthology.orchestra.run.smithi083.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-04T15:55:46.347 INFO:teuthology.orchestra.run.smithi083.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-04T15:55:46.347 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-03-04T15:55:46.347 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T15:55:46.368 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph 2024-03-04T15:55:46.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph 2024-03-04T15:55:46.459 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 777 /etc/ceph 2024-03-04T15:55:46.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /etc/ceph 2024-03-04T15:55:46.529 INFO:tasks.cephadm:Writing seed config... 2024-03-04T15:55:46.530 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-04T15:55:46.530 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-04T15:55:46.531 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-04T15:55:46.531 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-03-04T15:55:46.531 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-04T15:55:46.531 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-04T15:55:46.531 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-04T15:55:46.531 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-04T15:55:46.531 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-04T15:55:46.532 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-04T15:55:46.532 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-04T15:55:46.547 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 98ffad3c-da3f-11ee-95c6-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-04T15:55:46.547 DEBUG:teuthology.orchestra.run.smithi083:mon.a> sudo journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mon.a.service 2024-03-04T15:55:46.590 DEBUG:teuthology.orchestra.run.smithi083:mgr.y> sudo journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mgr.y.service 2024-03-04T15:55:46.633 INFO:tasks.cephadm:Bootstrapping... 2024-03-04T15:55:46.634 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.83 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:55:46.830 INFO:teuthology.orchestra.run.smithi083.stderr:-------------------------------------------------------------------------------- 2024-03-04T15:55:46.830 INFO:teuthology.orchestra.run.smithi083.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '98ffad3c-da3f-11ee-95c6-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.83', '--skip-admin-label'] 2024-03-04T15:55:46.852 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: 4.9.3-dev 2024-03-04T15:55:46.854 INFO:teuthology.orchestra.run.smithi083.stderr:Verifying podman|docker is present... 2024-03-04T15:55:46.876 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: 4.9.3-dev 2024-03-04T15:55:46.877 INFO:teuthology.orchestra.run.smithi083.stderr:Verifying lvm2 is present... 2024-03-04T15:55:46.877 INFO:teuthology.orchestra.run.smithi083.stderr:Verifying time synchronization is in place... 2024-03-04T15:55:46.884 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-04T15:55:46.890 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: inactive 2024-03-04T15:55:46.896 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: enabled 2024-03-04T15:55:46.902 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: active 2024-03-04T15:55:46.902 INFO:teuthology.orchestra.run.smithi083.stderr:Unit chronyd.service is enabled and running 2024-03-04T15:55:46.902 INFO:teuthology.orchestra.run.smithi083.stderr:Repeating the final host check... 2024-03-04T15:55:46.923 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: 4.9.3-dev 2024-03-04T15:55:46.925 INFO:teuthology.orchestra.run.smithi083.stderr:podman (/bin/podman) version 4.9.3 is present 2024-03-04T15:55:46.925 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl is present 2024-03-04T15:55:46.925 INFO:teuthology.orchestra.run.smithi083.stderr:lvcreate is present 2024-03-04T15:55:46.931 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-04T15:55:46.937 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: inactive 2024-03-04T15:55:46.943 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: enabled 2024-03-04T15:55:46.949 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: active 2024-03-04T15:55:46.950 INFO:teuthology.orchestra.run.smithi083.stderr:Unit chronyd.service is enabled and running 2024-03-04T15:55:46.950 INFO:teuthology.orchestra.run.smithi083.stderr:Host looks OK 2024-03-04T15:55:46.950 INFO:teuthology.orchestra.run.smithi083.stderr:Cluster fsid: 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:55:46.950 INFO:teuthology.orchestra.run.smithi083.stderr:Acquiring lock 140455480307616 on /run/cephadm/98ffad3c-da3f-11ee-95c6-87774f69a715.lock 2024-03-04T15:55:46.950 INFO:teuthology.orchestra.run.smithi083.stderr:Lock 140455480307616 acquired on /run/cephadm/98ffad3c-da3f-11ee-95c6-87774f69a715.lock 2024-03-04T15:55:46.951 INFO:teuthology.orchestra.run.smithi083.stderr:Verifying IP 172.21.15.83 port 3300 ... 2024-03-04T15:55:46.951 INFO:teuthology.orchestra.run.smithi083.stderr:Verifying IP 172.21.15.83 port 6789 ... 2024-03-04T15:55:46.951 INFO:teuthology.orchestra.run.smithi083.stderr:Base mon IP is 172.21.15.83, final addrv is [v2:172.21.15.83:3300,v1:172.21.15.83:6789] 2024-03-04T15:55:46.953 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.83 metric 100 2024-03-04T15:55:46.953 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.83 metric 100 2024-03-04T15:55:46.955 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-03-04T15:55:46.955 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-04T15:55:46.956 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1793sec hoplimit 64 pref medium 2024-03-04T15:55:46.958 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-04T15:55:46.959 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-03-04T15:55:46.959 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-04T15:55:46.959 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-04T15:55:46.959 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:ce91/64 scope link 2024-03-04T15:55:46.959 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-04T15:55:46.959 INFO:teuthology.orchestra.run.smithi083.stderr:Mon IP `172.21.15.83` is in CIDR network `172.21.0.0/20` 2024-03-04T15:55:46.959 INFO:teuthology.orchestra.run.smithi083.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-04T15:55:46.960 INFO:teuthology.orchestra.run.smithi083.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-04T15:55:47.061 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-03-04T15:55:47.518 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Getting image source signatures 2024-03-04T15:55:47.518 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-03-04T15:55:47.518 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-03-04T15:55:47.518 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-03-04T15:55:47.518 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-03-04T15:55:47.519 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-03-04T15:55:47.519 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-04T15:55:48.083 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Writing manifest to image destination 2024-03-04T15:55:48.236 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-04T15:55:48.675 INFO:teuthology.orchestra.run.smithi083.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-04T15:55:49.461 INFO:teuthology.orchestra.run.smithi083.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-04T15:55:49.461 INFO:teuthology.orchestra.run.smithi083.stderr:Extracting ceph user uid/gid from container image... 2024-03-04T15:55:49.860 INFO:teuthology.orchestra.run.smithi083.stderr:stat: 167 167 2024-03-04T15:55:50.382 INFO:teuthology.orchestra.run.smithi083.stderr:Creating initial keys... 2024-03-04T15:55:50.807 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-authtool: AQAG7+Vlj52RKxAAvJ/Q60BD38gT9qRR6ipE1Q== 2024-03-04T15:55:51.679 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-authtool: AQAH7+VlRRgOIxAAFPfr27jjUa2JrpLmOhiNBg== 2024-03-04T15:55:52.760 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-authtool: AQAI7+VloaQ3KBAATYdceIvDEz7q8Lu3MtEppA== 2024-03-04T15:55:54.303 INFO:teuthology.orchestra.run.smithi083.stderr:Creating initial monmap... 2024-03-04T15:55:57.259 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-04T15:55:57.259 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-03-04T15:55:57.260 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:55:57.260 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-04T15:55:57.810 INFO:teuthology.orchestra.run.smithi083.stderr:monmaptool for a [v2:172.21.15.83:3300,v1:172.21.15.83:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-04T15:55:57.810 INFO:teuthology.orchestra.run.smithi083.stderr:setting min_mon_release = octopus 2024-03-04T15:55:57.810 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/monmaptool: set fsid to 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:55:57.810 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-04T15:55:57.810 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-04T15:55:57.810 INFO:teuthology.orchestra.run.smithi083.stderr:Creating mon... 2024-03-04T15:55:58.795 INFO:teuthology.orchestra.run.smithi083.stderr:create mon.a on 2024-03-04T15:55:59.004 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-04T15:55:59.148 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-98ffad3c-da3f-11ee-95c6-87774f69a715.target → /etc/systemd/system/ceph-98ffad3c-da3f-11ee-95c6-87774f69a715.target. 2024-03-04T15:55:59.148 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-98ffad3c-da3f-11ee-95c6-87774f69a715.target → /etc/systemd/system/ceph-98ffad3c-da3f-11ee-95c6-87774f69a715.target. 2024-03-04T15:55:59.436 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Failed to reset failed state of unit ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mon.a.service: Unit ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mon.a.service not loaded. 2024-03-04T15:55:59.443 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Created symlink /etc/systemd/system/ceph-98ffad3c-da3f-11ee-95c6-87774f69a715.target.wants/ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mon.a.service → /etc/systemd/system/ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@.service. 2024-03-04T15:56:00.170 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:55:59 smithi083 podman[24365]: 2024-03-04 15:55:59.730623336 +0000 UTC m=+0.019000067 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-04T15:56:00.335 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: disabled 2024-03-04T15:56:00.341 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: inactive 2024-03-04T15:56:00.342 INFO:teuthology.orchestra.run.smithi083.stderr:firewalld.service is not enabled 2024-03-04T15:56:00.342 INFO:teuthology.orchestra.run.smithi083.stderr:Not possible to enable service . firewalld.service is not available 2024-03-04T15:56:00.342 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for mon to start... 2024-03-04T15:56:00.342 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for mon... 2024-03-04T15:56:00.467 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:00 smithi083 podman[24365]: 2024-03-04 15:56:00.237692898 +0000 UTC m=+0.526069629 container init 94e74a82ccb5d866f8e84cd9018a26cba4508ee8353987cb15d76b9dd34931cb (image=quay.io/ceph/ceph:v17.2.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, maintainer=Guillaume Abrioux , version=8, vendor=Red Hat, Inc., GIT_CLEAN=True, distribution-scope=public, GIT_BRANCH=HEAD, vcs-type=git, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-04T15:56:00.468 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:00 smithi083 podman[24365]: 2024-03-04 15:56:00.240670023 +0000 UTC m=+0.529046757 container start 94e74a82ccb5d866f8e84cd9018a26cba4508ee8353987cb15d76b9dd34931cb (image=quay.io/ceph/ceph:v17.2.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a, name=centos-stream, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, version=8, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, ceph=True, vcs-type=git, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream) 2024-03-04T15:56:00.468 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:00 smithi083 ceph-mon[24384]: mkfs 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:56:00.468 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:00 smithi083 ceph-mon[24384]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-04T15:56:00.468 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:00 smithi083 bash[24365]: 94e74a82ccb5d866f8e84cd9018a26cba4508ee8353987cb15d76b9dd34931cb 2024-03-04T15:56:00.468 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:00 smithi083 systemd[1]: Started Ceph mon.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T15:56:00.881 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: cluster: 2024-03-04T15:56:00.881 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: id: 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:56:00.881 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-03-04T15:56:00.882 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-03-04T15:56:00.882 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: services: 2024-03-04T15:56:00.882 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.576906s) 2024-03-04T15:56:00.882 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mgr: no daemons active 2024-03-04T15:56:00.882 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-03-04T15:56:00.882 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-03-04T15:56:00.882 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: data: 2024-03-04T15:56:00.882 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-03-04T15:56:00.882 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-03-04T15:56:00.882 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-03-04T15:56:00.882 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: pgs: 2024-03-04T15:56:00.882 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-03-04T15:56:01.437 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:01 smithi083 ceph-mon[24384]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-04T15:56:01.437 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:01 smithi083 ceph-mon[24384]: monmap e1: 1 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0]} 2024-03-04T15:56:01.437 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:01 smithi083 ceph-mon[24384]: fsmap 2024-03-04T15:56:01.438 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:01 smithi083 ceph-mon[24384]: osdmap e1: 0 total, 0 up, 0 in 2024-03-04T15:56:01.438 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:01 smithi083 ceph-mon[24384]: mgrmap e1: no daemons active 2024-03-04T15:56:01.438 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:01 smithi083 ceph-mon[24384]: from='client.? 172.21.15.83:0/435091960' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-04T15:56:01.440 INFO:teuthology.orchestra.run.smithi083.stderr:mon is available 2024-03-04T15:56:01.440 INFO:teuthology.orchestra.run.smithi083.stderr:Assimilating anything we can from ceph.conf... 2024-03-04T15:56:01.936 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-03-04T15:56:01.936 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: [global] 2024-03-04T15:56:01.936 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: fsid = 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:56:01.936 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.83:3300,v1:172.21.15.83:6789] 2024-03-04T15:56:01.936 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-04T15:56:01.936 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-04T15:56:01.936 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-04T15:56:01.936 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-04T15:56:01.937 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-03-04T15:56:01.937 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: [mgr] 2024-03-04T15:56:01.937 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-03-04T15:56:01.937 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-04T15:56:01.937 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-03-04T15:56:01.937 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: [osd] 2024-03-04T15:56:01.937 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-04T15:56:01.937 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-04T15:56:02.736 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:02 smithi083 ceph-mon[24384]: from='client.? 172.21.15.83:0/2479106953' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-04T15:56:02.736 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:02 smithi083 ceph-mon[24384]: from='client.? 172.21.15.83:0/2479106953' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-04T15:56:02.737 INFO:teuthology.orchestra.run.smithi083.stderr:Generating new minimal ceph.conf... 2024-03-04T15:56:03.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:03 smithi083 ceph-mon[24384]: from='client.? 172.21.15.83:0/763999327' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:56:03.772 INFO:teuthology.orchestra.run.smithi083.stderr:Restarting the monitor... 2024-03-04T15:56:04.222 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:03 smithi083 systemd[1]: Stopping Ceph mon.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T15:56:04.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:03 smithi083 bash[24616]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon.a" found: no such container 2024-03-04T15:56:04.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:03 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a[24380]: 2024-03-04T15:56:03.929+0000 7f8801caa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:56:04.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:03 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a[24380]: 2024-03-04T15:56:03.929+0000 7f8801caa700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-04T15:56:04.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:03 smithi083 podman[24627]: 2024-03-04 15:56:03.950177414 +0000 UTC m=+0.126195065 container died 94e74a82ccb5d866f8e84cd9018a26cba4508ee8353987cb15d76b9dd34931cb (image=quay.io/ceph/ceph:v17.2.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.19.8, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, distribution-scope=public, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-03-04T15:56:04.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:04 smithi083 podman[24627]: 2024-03-04 15:56:04.320538552 +0000 UTC m=+0.496556212 container cleanup 94e74a82ccb5d866f8e84cd9018a26cba4508ee8353987cb15d76b9dd34931cb (image=quay.io/ceph/ceph:v17.2.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, ceph=True, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, vcs-type=git, distribution-scope=public, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, build-date=2022-05-03T08:36:31.336870) 2024-03-04T15:56:04.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:04 smithi083 bash[24627]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a 2024-03-04T15:56:04.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:04 smithi083 bash[24653]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon.a" found: no such container 2024-03-04T15:56:04.906 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:04 smithi083 podman[24641]: 2024-03-04 15:56:04.563314073 +0000 UTC m=+0.611438877 container remove 94e74a82ccb5d866f8e84cd9018a26cba4508ee8353987cb15d76b9dd34931cb (image=quay.io/ceph/ceph:v17.2.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a, ceph=True, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, distribution-scope=public, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, vcs-type=git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-03-04T15:56:04.906 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:04 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-04T15:56:04.906 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:04 smithi083 systemd[1]: Stopped Ceph mon.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T15:56:04.906 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:04 smithi083 systemd[1]: Starting Ceph mon.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T15:56:04.906 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:04 smithi083 podman[24719]: 2024-03-04 15:56:04.697525831 +0000 UTC m=+0.018318933 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-04T15:56:04.906 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:04 smithi083 podman[24719]: 2024-03-04 15:56:04.79717209 +0000 UTC m=+0.117965186 container create d2ba878ed364f67a2e89a9d4c420c99fef93063d13a2339c7871b8c0076cf018 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.buildah.version=1.19.8, name=centos-stream, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-04T15:56:05.079 INFO:teuthology.orchestra.run.smithi083.stderr:Setting mon public_network to 172.21.0.0/20 2024-03-04T15:56:05.203 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:04 smithi083 podman[24719]: 2024-03-04 15:56:04.991236243 +0000 UTC m=+0.312029348 container init d2ba878ed364f67a2e89a9d4c420c99fef93063d13a2339c7871b8c0076cf018 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, release=754, build-date=2022-05-03T08:36:31.336870, vcs-type=git, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, architecture=x86_64, io.buildah.version=1.19.8, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, name=centos-stream) 2024-03-04T15:56:05.203 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:04 smithi083 podman[24719]: 2024-03-04 15:56:04.994042433 +0000 UTC m=+0.314835534 container start d2ba878ed364f67a2e89a9d4c420c99fef93063d13a2339c7871b8c0076cf018 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a, maintainer=Guillaume Abrioux , io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, version=8, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, vcs-type=git, architecture=x86_64, name=centos-stream, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, ceph=True) 2024-03-04T15:56:05.203 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: set uid:gid to 167:167 (ceph:ceph) 2024-03-04T15:56:05.203 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-04T15:56:05.203 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: pidfile_write: ignore empty --pid-file 2024-03-04T15:56:05.203 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: load: jerasure load: lrc 2024-03-04T15:56:05.203 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: RocksDB version: 6.15.5 2024-03-04T15:56:05.204 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-04T15:56:05.204 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Compile date Apr 18 2022 2024-03-04T15:56:05.204 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: DB SUMMARY 2024-03-04T15:56:05.204 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: DB Session ID: VRPUR5762EGK1GU5BCQ4 2024-03-04T15:56:05.204 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: CURRENT file: CURRENT 2024-03-04T15:56:05.204 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: IDENTITY file: IDENTITY 2024-03-04T15:56:05.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-04T15:56:05.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-04T15:56:05.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82629 ; 2024-03-04T15:56:05.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.error_if_exists: 0 2024-03-04T15:56:05.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.create_if_missing: 0 2024-03-04T15:56:05.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.paranoid_checks: 1 2024-03-04T15:56:05.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-04T15:56:05.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.env: 0x560b44ec6860 2024-03-04T15:56:05.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.fs: Posix File System 2024-03-04T15:56:05.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.info_log: 0x560b45b41dc0 2024-03-04T15:56:05.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_file_opening_threads: 16 2024-03-04T15:56:05.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.statistics: (nil) 2024-03-04T15:56:05.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.use_fsync: 0 2024-03-04T15:56:05.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_log_file_size: 0 2024-03-04T15:56:05.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-04T15:56:05.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-04T15:56:05.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.keep_log_file_num: 1000 2024-03-04T15:56:05.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.recycle_log_file_num: 0 2024-03-04T15:56:05.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.allow_fallocate: 1 2024-03-04T15:56:05.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.allow_mmap_reads: 0 2024-03-04T15:56:05.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.allow_mmap_writes: 0 2024-03-04T15:56:05.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.use_direct_reads: 0 2024-03-04T15:56:05.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-04T15:56:05.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.create_missing_column_families: 0 2024-03-04T15:56:05.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.db_log_dir: 2024-03-04T15:56:05.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-03-04T15:56:05.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-04T15:56:05.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-04T15:56:05.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-04T15:56:05.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-04T15:56:05.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-04T15:56:05.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-04T15:56:05.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.advise_random_on_open: 1 2024-03-04T15:56:05.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.db_write_buffer_size: 0 2024-03-04T15:56:05.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.write_buffer_manager: 0x560b45c322a0 2024-03-04T15:56:05.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-04T15:56:05.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-04T15:56:05.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-04T15:56:05.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-04T15:56:05.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.rate_limiter: (nil) 2024-03-04T15:56:05.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-04T15:56:05.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.wal_recovery_mode: 2 2024-03-04T15:56:05.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.enable_thread_tracking: 0 2024-03-04T15:56:05.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.enable_pipelined_write: 0 2024-03-04T15:56:05.208 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.unordered_write: 0 2024-03-04T15:56:05.208 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-04T15:56:05.208 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-04T15:56:05.208 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-04T15:56:05.208 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-04T15:56:05.208 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.row_cache: None 2024-03-04T15:56:05.208 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.wal_filter: None 2024-03-04T15:56:05.208 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-04T15:56:05.208 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.allow_ingest_behind: 0 2024-03-04T15:56:05.208 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.preserve_deletes: 0 2024-03-04T15:56:05.208 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.two_write_queues: 0 2024-03-04T15:56:05.208 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.manual_wal_flush: 0 2024-03-04T15:56:05.208 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.atomic_flush: 0 2024-03-04T15:56:05.208 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-04T15:56:05.208 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-04T15:56:05.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-04T15:56:05.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.log_readahead_size: 0 2024-03-04T15:56:05.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-04T15:56:05.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.best_efforts_recovery: 0 2024-03-04T15:56:05.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-04T15:56:05.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-04T15:56:05.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.allow_data_in_errors: 0 2024-03-04T15:56:05.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.db_host_id: __hostname__ 2024-03-04T15:56:05.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_background_jobs: 2 2024-03-04T15:56:05.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_background_compactions: -1 2024-03-04T15:56:05.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_subcompactions: 1 2024-03-04T15:56:05.210 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-04T15:56:05.210 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-04T15:56:05.210 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-04T15:56:05.210 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_total_wal_size: 0 2024-03-04T15:56:05.210 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-04T15:56:05.210 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-04T15:56:05.210 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-04T15:56:05.210 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-04T15:56:05.210 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_open_files: -1 2024-03-04T15:56:05.210 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.bytes_per_sync: 0 2024-03-04T15:56:05.210 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-04T15:56:05.211 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-04T15:56:05.211 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compaction_readahead_size: 0 2024-03-04T15:56:05.211 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_background_flushes: -1 2024-03-04T15:56:05.211 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Compression algorithms supported: 2024-03-04T15:56:05.211 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-04T15:56:05.211 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: kZSTD supported: 0 2024-03-04T15:56:05.211 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: kXpressCompression supported: 0 2024-03-04T15:56:05.211 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: kLZ4HCCompression supported: 1 2024-03-04T15:56:05.211 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: kLZ4Compression supported: 1 2024-03-04T15:56:05.211 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: kBZip2Compression supported: 0 2024-03-04T15:56:05.211 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: kZlibCompression supported: 1 2024-03-04T15:56:05.211 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: kSnappyCompression supported: 1 2024-03-04T15:56:05.211 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-04T15:56:05.211 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-03-04T15:56:05.211 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-04T15:56:05.212 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-04T15:56:05.212 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.merge_operator: 2024-03-04T15:56:05.212 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compaction_filter: None 2024-03-04T15:56:05.212 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compaction_filter_factory: None 2024-03-04T15:56:05.212 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.sst_partitioner_factory: None 2024-03-04T15:56:05.212 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-04T15:56:05.212 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-04T15:56:05.212 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560b45b0dd00) 2024-03-04T15:56:05.212 INFO:journalctl@ceph.mon.a.smithi083.stdout: cache_index_and_filter_blocks: 1 2024-03-04T15:56:05.213 INFO:journalctl@ceph.mon.a.smithi083.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-04T15:56:05.213 INFO:journalctl@ceph.mon.a.smithi083.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-04T15:56:05.213 INFO:journalctl@ceph.mon.a.smithi083.stdout: pin_top_level_index_and_filter: 1 2024-03-04T15:56:05.213 INFO:journalctl@ceph.mon.a.smithi083.stdout: index_type: 0 2024-03-04T15:56:05.213 INFO:journalctl@ceph.mon.a.smithi083.stdout: data_block_index_type: 0 2024-03-04T15:56:05.213 INFO:journalctl@ceph.mon.a.smithi083.stdout: index_shortening: 1 2024-03-04T15:56:05.213 INFO:journalctl@ceph.mon.a.smithi083.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-04T15:56:05.213 INFO:journalctl@ceph.mon.a.smithi083.stdout: hash_index_allow_collision: 1 2024-03-04T15:56:05.213 INFO:journalctl@ceph.mon.a.smithi083.stdout: checksum: 1 2024-03-04T15:56:05.213 INFO:journalctl@ceph.mon.a.smithi083.stdout: no_block_cache: 0 2024-03-04T15:56:05.213 INFO:journalctl@ceph.mon.a.smithi083.stdout: block_cache: 0x560b45b78170 2024-03-04T15:56:05.213 INFO:journalctl@ceph.mon.a.smithi083.stdout: block_cache_name: BinnedLRUCache 2024-03-04T15:56:05.215 INFO:journalctl@ceph.mon.a.smithi083.stdout: block_cache_options: 2024-03-04T15:56:05.215 INFO:journalctl@ceph.mon.a.smithi083.stdout: capacity : 536870912 2024-03-04T15:56:05.215 INFO:journalctl@ceph.mon.a.smithi083.stdout: num_shard_bits : 4 2024-03-04T15:56:05.215 INFO:journalctl@ceph.mon.a.smithi083.stdout: strict_capacity_limit : 0 2024-03-04T15:56:05.215 INFO:journalctl@ceph.mon.a.smithi083.stdout: high_pri_pool_ratio: 0.000 2024-03-04T15:56:05.215 INFO:journalctl@ceph.mon.a.smithi083.stdout: block_cache_compressed: (nil) 2024-03-04T15:56:05.215 INFO:journalctl@ceph.mon.a.smithi083.stdout: persistent_cache: (nil) 2024-03-04T15:56:05.215 INFO:journalctl@ceph.mon.a.smithi083.stdout: block_size: 4096 2024-03-04T15:56:05.215 INFO:journalctl@ceph.mon.a.smithi083.stdout: block_size_deviation: 10 2024-03-04T15:56:05.215 INFO:journalctl@ceph.mon.a.smithi083.stdout: block_restart_interval: 16 2024-03-04T15:56:05.215 INFO:journalctl@ceph.mon.a.smithi083.stdout: index_block_restart_interval: 1 2024-03-04T15:56:05.215 INFO:journalctl@ceph.mon.a.smithi083.stdout: metadata_block_size: 4096 2024-03-04T15:56:05.216 INFO:journalctl@ceph.mon.a.smithi083.stdout: partition_filters: 0 2024-03-04T15:56:05.216 INFO:journalctl@ceph.mon.a.smithi083.stdout: use_delta_encoding: 1 2024-03-04T15:56:05.216 INFO:journalctl@ceph.mon.a.smithi083.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-04T15:56:05.216 INFO:journalctl@ceph.mon.a.smithi083.stdout: whole_key_filtering: 1 2024-03-04T15:56:05.216 INFO:journalctl@ceph.mon.a.smithi083.stdout: verify_compression: 0 2024-03-04T15:56:05.216 INFO:journalctl@ceph.mon.a.smithi083.stdout: read_amp_bytes_per_bit: 0 2024-03-04T15:56:05.216 INFO:journalctl@ceph.mon.a.smithi083.stdout: format_version: 4 2024-03-04T15:56:05.216 INFO:journalctl@ceph.mon.a.smithi083.stdout: enable_index_compression: 1 2024-03-04T15:56:05.216 INFO:journalctl@ceph.mon.a.smithi083.stdout: block_align: 0 2024-03-04T15:56:05.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.write_buffer_size: 33554432 2024-03-04T15:56:05.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_write_buffer_number: 2 2024-03-04T15:56:05.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compression: NoCompression 2024-03-04T15:56:05.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.bottommost_compression: Disabled 2024-03-04T15:56:05.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.prefix_extractor: nullptr 2024-03-04T15:56:05.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-04T15:56:05.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.num_levels: 7 2024-03-04T15:56:05.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-04T15:56:05.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-04T15:56:05.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-04T15:56:05.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-04T15:56:05.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-04T15:56:05.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-04T15:56:05.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-04T15:56:05.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:56:05.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-04T15:56:05.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-04T15:56:05.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-04T15:56:05.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compression_opts.level: 32767 2024-03-04T15:56:05.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compression_opts.strategy: 0 2024-03-04T15:56:05.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-04T15:56:05.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:56:05.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-04T15:56:05.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compression_opts.enabled: false 2024-03-04T15:56:05.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-04T15:56:05.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-04T15:56:05.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-04T15:56:05.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.target_file_size_base: 67108864 2024-03-04T15:56:05.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-04T15:56:05.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-04T15:56:05.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-04T15:56:05.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-04T15:56:05.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-04T15:56:05.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-04T15:56:05.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-04T15:56:05.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-04T15:56:05.220 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-04T15:56:05.220 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-04T15:56:05.220 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-04T15:56:05.220 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-04T15:56:05.220 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-04T15:56:05.220 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.arena_block_size: 4194304 2024-03-04T15:56:05.220 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-04T15:56:05.220 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-04T15:56:05.220 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-04T15:56:05.220 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.disable_auto_compactions: 0 2024-03-04T15:56:05.221 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-04T15:56:05.221 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-04T15:56:05.221 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-04T15:56:05.221 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-04T15:56:05.221 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-04T15:56:05.221 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-04T15:56:05.221 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-04T15:56:05.221 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-04T15:56:05.221 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-04T15:56:05.221 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-04T15:56:05.222 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.table_properties_collectors: 2024-03-04T15:56:05.222 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.inplace_update_support: 0 2024-03-04T15:56:05.222 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-04T15:56:05.222 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-04T15:56:05.222 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-04T15:56:05.222 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-04T15:56:05.222 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.bloom_locality: 0 2024-03-04T15:56:05.222 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.max_successive_merges: 0 2024-03-04T15:56:05.222 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-04T15:56:05.222 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.paranoid_file_checks: 0 2024-03-04T15:56:05.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.force_consistency_checks: 1 2024-03-04T15:56:05.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.report_bg_io_stats: 0 2024-03-04T15:56:05.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.ttl: 2592000 2024-03-04T15:56:05.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-04T15:56:05.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.enable_blob_files: false 2024-03-04T15:56:05.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.min_blob_size: 0 2024-03-04T15:56:05.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.blob_file_size: 268435456 2024-03-04T15:56:05.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-04T15:56:05.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-04T15:56:05.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-04T15:56:05.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-04T15:56:05.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-03-04T15:56:05.224 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-03-04T15:56:05.224 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709567765018857, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-04T15:56:05.224 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-03-04T15:56:05.224 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-04T15:56:05.224 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709567765019540, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79179, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77442, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71751, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1709567765, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "1a4d993a-4266-494c-a91d-b730cf20813b", "db_session_id": "VRPUR5762EGK1GU5BCQ4"}} 2024-03-04T15:56:05.224 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-03-04T15:56:05.224 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709567765019846, "job": 1, "event": "recovery_finished"} 2024-03-04T15:56:05.224 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-04T15:56:05.224 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x560b45b5ea80 2024-03-04T15:56:05.224 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: DB pointer 0x560b45b6e000 2024-03-04T15:56:05.224 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-04T15:56:05.224 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-04T15:56:05.224 INFO:journalctl@ceph.mon.a.smithi083.stdout: ** DB Stats ** 2024-03-04T15:56:05.225 INFO:journalctl@ceph.mon.a.smithi083.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:56:05.225 INFO:journalctl@ceph.mon.a.smithi083.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-04T15:56:05.225 INFO:journalctl@ceph.mon.a.smithi083.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:56:05.225 INFO:journalctl@ceph.mon.a.smithi083.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:56:05.225 INFO:journalctl@ceph.mon.a.smithi083.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-04T15:56:05.225 INFO:journalctl@ceph.mon.a.smithi083.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-04T15:56:05.225 INFO:journalctl@ceph.mon.a.smithi083.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:56:05.225 INFO:journalctl@ceph.mon.a.smithi083.stdout: 2024-03-04T15:56:05.225 INFO:journalctl@ceph.mon.a.smithi083.stdout: ** Compaction Stats [default] ** 2024-03-04T15:56:05.225 INFO:journalctl@ceph.mon.a.smithi083.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-04T15:56:05.225 INFO:journalctl@ceph.mon.a.smithi083.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T15:56:05.225 INFO:journalctl@ceph.mon.a.smithi083.stdout: L0 2/0 79.08 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.4 0.00 0.00 1 0.000 0 0 2024-03-04T15:56:05.225 INFO:journalctl@ceph.mon.a.smithi083.stdout: Sum 2/0 79.08 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.4 0.00 0.00 1 0.000 0 0 2024-03-04T15:56:05.226 INFO:journalctl@ceph.mon.a.smithi083.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.4 0.00 0.00 1 0.000 0 0 2024-03-04T15:56:05.226 INFO:journalctl@ceph.mon.a.smithi083.stdout: 2024-03-04T15:56:05.226 INFO:journalctl@ceph.mon.a.smithi083.stdout: ** Compaction Stats [default] ** 2024-03-04T15:56:05.226 INFO:journalctl@ceph.mon.a.smithi083.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-04T15:56:05.226 INFO:journalctl@ceph.mon.a.smithi083.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T15:56:05.226 INFO:journalctl@ceph.mon.a.smithi083.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 167.4 0.00 0.00 1 0.000 0 0 2024-03-04T15:56:05.226 INFO:journalctl@ceph.mon.a.smithi083.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:56:05.226 INFO:journalctl@ceph.mon.a.smithi083.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-04T15:56:05.226 INFO:journalctl@ceph.mon.a.smithi083.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T15:56:05.226 INFO:journalctl@ceph.mon.a.smithi083.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T15:56:05.226 INFO:journalctl@ceph.mon.a.smithi083.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T15:56:05.226 INFO:journalctl@ceph.mon.a.smithi083.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T15:56:05.226 INFO:journalctl@ceph.mon.a.smithi083.stdout: Cumulative compaction: 0.00 GB write, 26.88 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:56:05.226 INFO:journalctl@ceph.mon.a.smithi083.stdout: Interval compaction: 0.00 GB write, 26.88 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:56:05.227 INFO:journalctl@ceph.mon.a.smithi083.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T15:56:05.227 INFO:journalctl@ceph.mon.a.smithi083.stdout: 2024-03-04T15:56:05.227 INFO:journalctl@ceph.mon.a.smithi083.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T15:56:05.227 INFO:journalctl@ceph.mon.a.smithi083.stdout: 2024-03-04T15:56:05.227 INFO:journalctl@ceph.mon.a.smithi083.stdout: ** Compaction Stats [default] ** 2024-03-04T15:56:05.227 INFO:journalctl@ceph.mon.a.smithi083.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-04T15:56:05.227 INFO:journalctl@ceph.mon.a.smithi083.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T15:56:05.227 INFO:journalctl@ceph.mon.a.smithi083.stdout: L0 2/0 79.08 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.4 0.00 0.00 1 0.000 0 0 2024-03-04T15:56:05.227 INFO:journalctl@ceph.mon.a.smithi083.stdout: Sum 2/0 79.08 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.4 0.00 0.00 1 0.000 0 0 2024-03-04T15:56:05.227 INFO:journalctl@ceph.mon.a.smithi083.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-04T15:56:05.227 INFO:journalctl@ceph.mon.a.smithi083.stdout: 2024-03-04T15:56:05.227 INFO:journalctl@ceph.mon.a.smithi083.stdout: ** Compaction Stats [default] ** 2024-03-04T15:56:05.227 INFO:journalctl@ceph.mon.a.smithi083.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-04T15:56:05.228 INFO:journalctl@ceph.mon.a.smithi083.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T15:56:05.228 INFO:journalctl@ceph.mon.a.smithi083.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 167.4 0.00 0.00 1 0.000 0 0 2024-03-04T15:56:05.228 INFO:journalctl@ceph.mon.a.smithi083.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:56:05.228 INFO:journalctl@ceph.mon.a.smithi083.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-04T15:56:05.228 INFO:journalctl@ceph.mon.a.smithi083.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T15:56:05.228 INFO:journalctl@ceph.mon.a.smithi083.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T15:56:05.228 INFO:journalctl@ceph.mon.a.smithi083.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T15:56:05.228 INFO:journalctl@ceph.mon.a.smithi083.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T15:56:05.228 INFO:journalctl@ceph.mon.a.smithi083.stdout: Cumulative compaction: 0.00 GB write, 26.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:56:05.228 INFO:journalctl@ceph.mon.a.smithi083.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:56:05.228 INFO:journalctl@ceph.mon.a.smithi083.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T15:56:05.228 INFO:journalctl@ceph.mon.a.smithi083.stdout: 2024-03-04T15:56:05.228 INFO:journalctl@ceph.mon.a.smithi083.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T15:56:05.228 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: starting mon.a rank 0 at public addrs [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] at bind addrs [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:56:05.229 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: mon.a@-1(???) e1 preinit fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:56:05.229 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: mon.a@-1(???).mds e1 new map 2024-03-04T15:56:05.229 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: mon.a@-1(???).mds e1 print_map 2024-03-04T15:56:05.229 INFO:journalctl@ceph.mon.a.smithi083.stdout: e1 2024-03-04T15:56:05.229 INFO:journalctl@ceph.mon.a.smithi083.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T15:56:05.229 INFO:journalctl@ceph.mon.a.smithi083.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T15:56:05.229 INFO:journalctl@ceph.mon.a.smithi083.stdout: legacy client fscid: -1 2024-03-04T15:56:05.229 INFO:journalctl@ceph.mon.a.smithi083.stdout: 2024-03-04T15:56:05.229 INFO:journalctl@ceph.mon.a.smithi083.stdout: No filesystems configured 2024-03-04T15:56:05.229 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-04T15:56:05.229 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:56:05.229 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:56:05.229 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:56:05.230 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-04T15:56:05.230 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 bash[24719]: d2ba878ed364f67a2e89a9d4c420c99fef93063d13a2339c7871b8c0076cf018 2024-03-04T15:56:05.230 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 systemd[1]: Started Ceph mon.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T15:56:05.230 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-04T15:56:05.230 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: monmap e1: 1 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0]} 2024-03-04T15:56:05.230 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: fsmap 2024-03-04T15:56:05.230 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: osdmap e1: 0 total, 0 up, 0 in 2024-03-04T15:56:05.230 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:05 smithi083 ceph-mon[24738]: mgrmap e1: no daemons active 2024-03-04T15:56:06.186 INFO:teuthology.orchestra.run.smithi083.stderr:Wrote config to /etc/ceph/ceph.conf 2024-03-04T15:56:06.186 INFO:teuthology.orchestra.run.smithi083.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:56:06.186 INFO:teuthology.orchestra.run.smithi083.stderr:Creating mgr... 2024-03-04T15:56:06.186 INFO:teuthology.orchestra.run.smithi083.stderr:Verifying port 9283 ... 2024-03-04T15:56:06.355 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Failed to reset failed state of unit ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mgr.y.service: Unit ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mgr.y.service not loaded. 2024-03-04T15:56:06.362 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Created symlink /etc/systemd/system/ceph-98ffad3c-da3f-11ee-95c6-87774f69a715.target.wants/ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@.service. 2024-03-04T15:56:06.848 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:06 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/378356737' entity='client.admin' 2024-03-04T15:56:07.239 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:06 smithi083 podman[24956]: 2024-03-04 15:56:06.946128572 +0000 UTC m=+0.321567890 container start 5af09a7ff74e8e728272937bd2ee8c5ec835d53e1bf1b8d0a81e07318467092e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, distribution-scope=public, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, maintainer=Guillaume Abrioux , name=centos-stream, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-04T15:56:07.386 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: disabled 2024-03-04T15:56:07.393 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: inactive 2024-03-04T15:56:07.394 INFO:teuthology.orchestra.run.smithi083.stderr:firewalld.service is not enabled 2024-03-04T15:56:07.394 INFO:teuthology.orchestra.run.smithi083.stderr:Not possible to enable service . firewalld.service is not available 2024-03-04T15:56:07.404 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: disabled 2024-03-04T15:56:07.412 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: inactive 2024-03-04T15:56:07.413 INFO:teuthology.orchestra.run.smithi083.stderr:firewalld.service is not enabled 2024-03-04T15:56:07.413 INFO:teuthology.orchestra.run.smithi083.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-04T15:56:07.413 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for mgr to start... 2024-03-04T15:56:07.413 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for mgr... 2024-03-04T15:56:07.646 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:07 smithi083 bash[24956]: 5af09a7ff74e8e728272937bd2ee8c5ec835d53e1bf1b8d0a81e07318467092e 2024-03-04T15:56:07.647 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:07 smithi083 systemd[1]: Started Ceph mgr.y for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T15:56:07.647 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:07 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:07.429+0000 7f2249e3c000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:56:07.647 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:07 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:07.521+0000 7f2249e3c000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:56:07.920 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-03-04T15:56:07.920 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-03-04T15:56:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "fsid": "98ffad3c-da3f-11ee-95c6-87774f69a715", 2024-03-04T15:56:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "health": { 2024-03-04T15:56:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-04T15:56:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "checks": {}, 2024-03-04T15:56:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mutes": [] 2024-03-04T15:56:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-04T15:56:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum": [ 2024-03-04T15:56:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 0 2024-03-04T15:56:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-03-04T15:56:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-04T15:56:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "a" 2024-03-04T15:56:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-03-04T15:56:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-03-04T15:56:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "monmap": { 2024-03-04T15:56:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-04T15:56:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-04T15:56:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-04T15:56:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osdmap": { 2024-03-04T15:56:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-04T15:56:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-04T15:56:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-04T15:56:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-04T15:56:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-04T15:56:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-04T15:56:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-04T15:56:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "pgmap": { 2024-03-04T15:56:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-04T15:56:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-04T15:56:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-04T15:56:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-04T15:56:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-04T15:56:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-04T15:56:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-04T15:56:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-04T15:56:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "fsmap": { 2024-03-04T15:56:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-04T15:56:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-04T15:56:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-04T15:56:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-04T15:56:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "available": false, 2024-03-04T15:56:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-04T15:56:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "modules": [ 2024-03-04T15:56:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "iostat", 2024-03-04T15:56:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "nfs", 2024-03-04T15:56:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "restful" 2024-03-04T15:56:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-03-04T15:56:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "services": {} 2024-03-04T15:56:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "servicemap": { 2024-03-04T15:56:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-04T15:56:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "modified": "2024-03-04T15:56:00.303065+0000", 2024-03-04T15:56:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "services": {} 2024-03-04T15:56:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-04T15:56:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-03-04T15:56:08.229 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:07 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2190256017' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T15:56:08.229 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:08 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:08.140+0000 7f2249e3c000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:56:08.230 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:08 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:08.227+0000 7f2249e3c000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:56:08.489 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:08 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:08.312+0000 7f2249e3c000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:56:08.489 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:08 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:08.459+0000 7f2249e3c000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:56:08.571 INFO:teuthology.orchestra.run.smithi083.stderr:mgr not available, waiting (1/15)... 2024-03-04T15:56:08.893 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:08 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:08.540+0000 7f2249e3c000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:56:08.893 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:08 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:08.649+0000 7f2249e3c000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:56:08.893 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:08 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:08.731+0000 7f2249e3c000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:56:08.893 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:08 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:08.890+0000 7f2249e3c000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:56:09.239 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:08 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:08.974+0000 7f2249e3c000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:56:10.312 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:10 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:10.000+0000 7f2249e3c000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:56:10.313 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:10 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:10.126+0000 7f2249e3c000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:56:10.313 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:10 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:10.310+0000 7f2249e3c000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:56:10.700 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:10 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:10.394+0000 7f2249e3c000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:56:10.700 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:10 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:10.525+0000 7f2249e3c000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:56:10.989 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:10 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:10.856+0000 7f2249e3c000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:56:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-03-04T15:56:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-03-04T15:56:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "fsid": "98ffad3c-da3f-11ee-95c6-87774f69a715", 2024-03-04T15:56:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "health": { 2024-03-04T15:56:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-04T15:56:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "checks": {}, 2024-03-04T15:56:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mutes": [] 2024-03-04T15:56:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-04T15:56:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum": [ 2024-03-04T15:56:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 0 2024-03-04T15:56:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-03-04T15:56:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-04T15:56:11.107 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "a" 2024-03-04T15:56:11.107 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-03-04T15:56:11.107 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-03-04T15:56:11.107 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "monmap": { 2024-03-04T15:56:11.107 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-04T15:56:11.107 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-04T15:56:11.107 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-04T15:56:11.107 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:11.107 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osdmap": { 2024-03-04T15:56:11.107 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-04T15:56:11.107 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-04T15:56:11.107 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-04T15:56:11.107 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-04T15:56:11.107 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-04T15:56:11.111 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-04T15:56:11.111 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-04T15:56:11.111 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:11.111 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "pgmap": { 2024-03-04T15:56:11.111 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-04T15:56:11.111 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-04T15:56:11.111 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-04T15:56:11.111 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-04T15:56:11.111 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-04T15:56:11.111 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-04T15:56:11.111 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-04T15:56:11.111 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-04T15:56:11.112 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:11.112 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "fsmap": { 2024-03-04T15:56:11.112 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-04T15:56:11.112 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-04T15:56:11.112 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-04T15:56:11.112 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:11.112 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-04T15:56:11.112 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "available": false, 2024-03-04T15:56:11.112 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-04T15:56:11.112 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "modules": [ 2024-03-04T15:56:11.112 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "iostat", 2024-03-04T15:56:11.112 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "nfs", 2024-03-04T15:56:11.112 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "restful" 2024-03-04T15:56:11.113 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-03-04T15:56:11.113 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "services": {} 2024-03-04T15:56:11.113 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:11.113 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "servicemap": { 2024-03-04T15:56:11.113 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-04T15:56:11.113 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "modified": "2024-03-04T15:56:00.303065+0000", 2024-03-04T15:56:11.113 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "services": {} 2024-03-04T15:56:11.113 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:11.113 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-04T15:56:11.113 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-03-04T15:56:11.346 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:11 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/4150408176' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T15:56:11.347 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:11 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:11.079+0000 7f2249e3c000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:56:11.347 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:11 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:11.172+0000 7f2249e3c000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:56:11.347 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:11 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:11.344+0000 7f2249e3c000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:56:11.619 INFO:teuthology.orchestra.run.smithi083.stderr:mgr not available, waiting (2/15)... 2024-03-04T15:56:12.540 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:12 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:12.252+0000 7f2249e3c000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:56:12.540 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:12 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:12.538+0000 7f2249e3c000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:56:12.846 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:12 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:12.630+0000 7f2249e3c000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:56:12.846 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:12 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:12.777+0000 7f2249e3c000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:56:13.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:12 smithi083 ceph-mon[24738]: Activating manager daemon y 2024-03-04T15:56:13.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:12 smithi083 ceph-mon[24738]: mgrmap e2: y(active, starting, since 0.00274265s) 2024-03-04T15:56:13.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:12 smithi083 ceph-mon[24738]: from='mgr.14100 172.21.15.83:0/3009134125' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:56:13.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:12 smithi083 ceph-mon[24738]: from='mgr.14100 172.21.15.83:0/3009134125' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:56:13.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:12 smithi083 ceph-mon[24738]: from='mgr.14100 172.21.15.83:0/3009134125' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:56:13.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:12 smithi083 ceph-mon[24738]: from='mgr.14100 172.21.15.83:0/3009134125' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:56:13.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:12 smithi083 ceph-mon[24738]: from='mgr.14100 172.21.15.83:0/3009134125' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T15:56:13.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:12 smithi083 ceph-mon[24738]: Manager daemon y is now available 2024-03-04T15:56:13.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:12 smithi083 ceph-mon[24738]: from='mgr.14100 172.21.15.83:0/3009134125' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:56:13.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:12 smithi083 ceph-mon[24738]: from='mgr.14100 172.21.15.83:0/3009134125' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:56:13.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:12 smithi083 ceph-mon[24738]: from='mgr.14100 172.21.15.83:0/3009134125' entity='mgr.y' 2024-03-04T15:56:13.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:12 smithi083 ceph-mon[24738]: from='mgr.14100 172.21.15.83:0/3009134125' entity='mgr.y' 2024-03-04T15:56:13.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:12 smithi083 ceph-mon[24738]: from='mgr.14100 172.21.15.83:0/3009134125' entity='mgr.y' 2024-03-04T15:56:14.332 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-03-04T15:56:14.332 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-03-04T15:56:14.332 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "fsid": "98ffad3c-da3f-11ee-95c6-87774f69a715", 2024-03-04T15:56:14.332 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "health": { 2024-03-04T15:56:14.333 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-04T15:56:14.333 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "checks": {}, 2024-03-04T15:56:14.333 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mutes": [] 2024-03-04T15:56:14.333 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:14.333 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-04T15:56:14.333 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum": [ 2024-03-04T15:56:14.333 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 0 2024-03-04T15:56:14.333 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-03-04T15:56:14.333 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-04T15:56:14.333 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "a" 2024-03-04T15:56:14.333 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-03-04T15:56:14.333 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-03-04T15:56:14.333 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "monmap": { 2024-03-04T15:56:14.333 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-04T15:56:14.333 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-04T15:56:14.334 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-04T15:56:14.334 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:14.334 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osdmap": { 2024-03-04T15:56:14.334 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-04T15:56:14.334 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-04T15:56:14.334 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-04T15:56:14.334 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-04T15:56:14.334 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-04T15:56:14.334 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-04T15:56:14.334 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-04T15:56:14.334 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:14.334 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "pgmap": { 2024-03-04T15:56:14.334 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-04T15:56:14.334 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-04T15:56:14.335 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-04T15:56:14.335 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-04T15:56:14.335 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-04T15:56:14.335 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-04T15:56:14.335 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-04T15:56:14.335 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-04T15:56:14.335 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:14.335 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "fsmap": { 2024-03-04T15:56:14.335 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-04T15:56:14.335 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-04T15:56:14.335 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-04T15:56:14.335 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:14.335 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-04T15:56:14.335 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "available": true, 2024-03-04T15:56:14.335 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-04T15:56:14.336 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "modules": [ 2024-03-04T15:56:14.336 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "iostat", 2024-03-04T15:56:14.336 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "nfs", 2024-03-04T15:56:14.336 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "restful" 2024-03-04T15:56:14.336 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-03-04T15:56:14.336 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "services": {} 2024-03-04T15:56:14.336 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:14.336 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "servicemap": { 2024-03-04T15:56:14.336 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-04T15:56:14.336 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "modified": "2024-03-04T15:56:00.303065+0000", 2024-03-04T15:56:14.336 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "services": {} 2024-03-04T15:56:14.336 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-03-04T15:56:14.336 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-04T15:56:14.336 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-03-04T15:56:14.925 INFO:teuthology.orchestra.run.smithi083.stderr:mgr is available 2024-03-04T15:56:15.049 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:14 smithi083 ceph-mon[24738]: mgrmap e3: y(active, since 1.00616s) 2024-03-04T15:56:15.049 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:14 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3938929767' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T15:56:15.515 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-03-04T15:56:15.515 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: [global] 2024-03-04T15:56:15.516 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: fsid = 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:56:15.516 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-04T15:56:15.516 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-04T15:56:15.516 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-04T15:56:15.516 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-04T15:56:15.516 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-03-04T15:56:15.516 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: [mgr] 2024-03-04T15:56:15.516 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-04T15:56:15.516 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-03-04T15:56:15.516 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: [osd] 2024-03-04T15:56:15.516 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-04T15:56:15.516 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-04T15:56:16.073 INFO:teuthology.orchestra.run.smithi083.stderr:Enabling cephadm module... 2024-03-04T15:56:16.172 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:16 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3128339700' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-04T15:56:16.172 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:16 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3128339700' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-04T15:56:16.172 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:16 smithi083 ceph-mon[24738]: mgrmap e4: y(active, since 3s) 2024-03-04T15:56:17.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:17 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2847714924' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-04T15:56:17.239 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:17 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ignoring --setuser ceph since I am not root 2024-03-04T15:56:17.239 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:17 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ignoring --setgroup ceph since I am not root 2024-03-04T15:56:17.620 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:17 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:17.525+0000 7f71cc66b000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:56:17.620 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:17 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:17.618+0000 7f71cc66b000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:56:18.193 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-03-04T15:56:18.193 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 5, 2024-03-04T15:56:18.193 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "available": true, 2024-03-04T15:56:18.193 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "active_name": "y", 2024-03-04T15:56:18.193 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-04T15:56:18.193 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-03-04T15:56:18.204 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:18 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2847714924' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-04T15:56:18.204 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:18 smithi083 ceph-mon[24738]: mgrmap e5: y(active, since 4s) 2024-03-04T15:56:18.489 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:18 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:18.224+0000 7f71cc66b000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:56:18.489 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:18 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:18.313+0000 7f71cc66b000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:56:18.490 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:18 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:18.395+0000 7f71cc66b000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:56:18.820 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:18 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:18.545+0000 7f71cc66b000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:56:18.820 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:18 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:18.626+0000 7f71cc66b000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:56:18.820 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:18 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:18.735+0000 7f71cc66b000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:56:18.821 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:18 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:18.818+0000 7f71cc66b000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:56:18.829 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for the mgr to restart... 2024-03-04T15:56:18.829 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for mgr epoch 5... 2024-03-04T15:56:19.074 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:19 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/481935701' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-04T15:56:19.075 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:18 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:18.983+0000 7f71cc66b000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:56:19.075 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:19 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:19.072+0000 7f71cc66b000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:56:20.419 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:20 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:20.098+0000 7f71cc66b000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:56:20.419 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:20 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:20.227+0000 7f71cc66b000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:56:20.419 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:20 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:20.417+0000 7f71cc66b000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:56:20.739 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:20 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:20.504+0000 7f71cc66b000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:56:20.739 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:20 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:20.638+0000 7f71cc66b000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:56:21.239 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:20 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:20.951+0000 7f71cc66b000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:56:21.239 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:21 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:21.171+0000 7f71cc66b000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:56:21.739 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:21 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:21.255+0000 7f71cc66b000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:56:21.739 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:21 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:21.424+0000 7f71cc66b000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:56:22.626 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:22 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:22.331+0000 7f71cc66b000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:56:22.626 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:22 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:22.624+0000 7f71cc66b000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:56:22.935 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:22 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:22.718+0000 7f71cc66b000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:56:22.936 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:22 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:22.866+0000 7f71cc66b000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:56:23.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:22 smithi083 ceph-mon[24738]: Active manager daemon y restarted 2024-03-04T15:56:23.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:22 smithi083 ceph-mon[24738]: Activating manager daemon y 2024-03-04T15:56:23.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:22 smithi083 ceph-mon[24738]: osdmap e2: 0 total, 0 up, 0 in 2024-03-04T15:56:23.937 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: [04/Mar/2024:15:56:23] ENGINE Bus STARTING 2024-03-04T15:56:23.937 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: [04/Mar/2024:15:56:23] ENGINE Serving on https://172.21.15.83:7150 2024-03-04T15:56:23.937 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: [04/Mar/2024:15:56:23] ENGINE Bus STARTED 2024-03-04T15:56:23.950 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-03-04T15:56:23.950 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-03-04T15:56:23.950 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "initialized": true 2024-03-04T15:56:23.950 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-03-04T15:56:24.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:23 smithi083 ceph-mon[24738]: mgrmap e6: y(active, starting, since 0.0527231s) 2024-03-04T15:56:24.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:23 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:56:24.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:23 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T15:56:24.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:23 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:56:24.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:23 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:56:24.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:23 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:56:24.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:23 smithi083 ceph-mon[24738]: Manager daemon y is now available 2024-03-04T15:56:24.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:23 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' 2024-03-04T15:56:24.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:23 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' 2024-03-04T15:56:24.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:23 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:56:24.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:23 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:56:24.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:23 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:56:24.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:23 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:56:24.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:23 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:56:24.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:23 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' 2024-03-04T15:56:24.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:23 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:56:24.517 INFO:teuthology.orchestra.run.smithi083.stderr:mgr epoch 5 is available 2024-03-04T15:56:24.518 INFO:teuthology.orchestra.run.smithi083.stderr:Setting orchestrator backend to cephadm... 2024-03-04T15:56:25.104 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:24 smithi083 ceph-mon[24738]: [04/Mar/2024:15:56:23] ENGINE Bus STARTING 2024-03-04T15:56:25.104 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:24 smithi083 ceph-mon[24738]: [04/Mar/2024:15:56:23] ENGINE Serving on https://172.21.15.83:7150 2024-03-04T15:56:25.104 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:24 smithi083 ceph-mon[24738]: [04/Mar/2024:15:56:23] ENGINE Bus STARTED 2024-03-04T15:56:25.104 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:24 smithi083 ceph-mon[24738]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-04T15:56:25.104 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:24 smithi083 ceph-mon[24738]: mgrmap e7: y(active, since 1.06157s) 2024-03-04T15:56:25.104 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:24 smithi083 ceph-mon[24738]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-04T15:56:26.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:25 smithi083 ceph-mon[24738]: mgrmap e8: y(active, since 2s) 2024-03-04T15:56:26.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:25 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' 2024-03-04T15:56:26.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:25 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:56:26.238 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: value unchanged 2024-03-04T15:56:26.755 INFO:teuthology.orchestra.run.smithi083.stderr:Generating ssh key... 2024-03-04T15:56:27.155 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:26 smithi083 ceph-mon[24738]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:56:28.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-mon[24738]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:56:28.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' 2024-03-04T15:56:28.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' 2024-03-04T15:56:28.240 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: Generating public/private rsa key pair. 2024-03-04T15:56:28.240 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: Your identification has been saved in /tmp/tmplst08243/key. 2024-03-04T15:56:28.240 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: Your public key has been saved in /tmp/tmplst08243/key.pub. 2024-03-04T15:56:28.240 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: The key fingerprint is: 2024-03-04T15:56:28.240 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: SHA256:u/MkVEzbFXFUnEHzeM06sKd6rR8AhOo88v4k6cRVr64 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:56:28.240 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: The key's randomart image is: 2024-03-04T15:56:28.240 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: +---[RSA 3072]----+ 2024-03-04T15:56:28.240 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: | .o =OB| 2024-03-04T15:56:28.240 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: | .+ o . ==| 2024-03-04T15:56:28.240 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: | . * o . =| 2024-03-04T15:56:28.241 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: | . o o o o | 2024-03-04T15:56:28.241 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: | o S + + | 2024-03-04T15:56:28.241 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: | ..++ . . + . | 2024-03-04T15:56:28.241 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: | o=.+ o ... | 2024-03-04T15:56:28.241 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: | o.o.= .. .. | 2024-03-04T15:56:28.241 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: | .o.E++..o. | 2024-03-04T15:56:28.241 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: +----[SHA256]-----+ 2024-03-04T15:56:29.064 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXi7O6XJJh4OL8sJZ6hxmAahkO7jy6aNf5uz1ATj5sqCR/JOGkAv3aG+P/XVxBcfWOuFJdKmC6fwx+kkTLCHKo1aeXXG+bz0AKiDDSrgsLQLLzpjQ6lMqeR/PYGvcZlm2dheKRnUSxEuA1Nzh35or6+sJLuXcxKTn/z3zFtSNRladawkHti815lHYYEDQfChqMwNPkfTjBSCTWZ8jA01ppuZXBkcV1Eqx4uVlPL+7NSY7JWNQTb6tKyEnFbnoQg0TU47YAnW/6vtEc+9YMmgT7CGIRP/tknDDQIMY75/xTQb5VkjIdiWAbTkShRpOeTzppFeX7CEOmqcmvslvHxbYojrtZf4c+7zIeNxiMIhk51MyNMZYKLnnBUzm6vSsnijQO4qn3eMZFDoBZx8dtkvRXq1jYEBbfoxllxftK4m7/lnaIQWyLIqKgnBUB3KiKZNQCPvV3mKutn1wzjPF7CbSBpTlBfUayx4po53r2afLugbrO21khhgEZppLUMTCTJiE= ceph-98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:56:29.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:28 smithi083 ceph-mon[24738]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:56:29.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:28 smithi083 ceph-mon[24738]: Generating ssh key... 2024-03-04T15:56:29.543 INFO:teuthology.orchestra.run.smithi083.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-04T15:56:29.543 INFO:teuthology.orchestra.run.smithi083.stderr:Adding key to root@localhost authorized_keys... 2024-03-04T15:56:29.544 INFO:teuthology.orchestra.run.smithi083.stderr:Adding host smithi083... 2024-03-04T15:56:30.163 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:29 smithi083 ceph-mon[24738]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:56:31.090 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Added host 'smithi083' with addr '172.21.15.83' 2024-03-04T15:56:31.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:31 smithi083 ceph-mon[24738]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi083", "addr": "172.21.15.83", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:56:31.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:31 smithi083 ceph-mon[24738]: Deploying cephadm binary to smithi083 2024-03-04T15:56:31.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:31 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' 2024-03-04T15:56:31.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:31 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:56:31.525 INFO:teuthology.orchestra.run.smithi083.stderr:Deploying unmanaged mon service... 2024-03-04T15:56:32.114 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Scheduled mon update... 2024-03-04T15:56:32.403 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:32 smithi083 ceph-mon[24738]: Added host smithi083 2024-03-04T15:56:32.403 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:32 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' 2024-03-04T15:56:32.706 INFO:teuthology.orchestra.run.smithi083.stderr:Deploying unmanaged mgr service... 2024-03-04T15:56:33.431 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:33 smithi083 ceph-mon[24738]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:56:33.432 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:33 smithi083 ceph-mon[24738]: Saving service mon spec with placement count:5 2024-03-04T15:56:33.523 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-03-04T15:56:34.853 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:34 smithi083 ceph-mon[24738]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:56:34.854 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:34 smithi083 ceph-mon[24738]: Saving service mgr spec with placement count:2 2024-03-04T15:56:34.854 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:34 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' 2024-03-04T15:56:34.854 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:34 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' 2024-03-04T15:56:36.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:35 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' 2024-03-04T15:56:36.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:35 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3086011814' entity='client.admin' 2024-03-04T15:56:37.748 INFO:teuthology.orchestra.run.smithi083.stderr:Enabling the dashboard module... 2024-03-04T15:56:37.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:37 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' 2024-03-04T15:56:37.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:37 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3462529480' entity='client.admin' 2024-03-04T15:56:37.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:37 smithi083 ceph-mon[24738]: from='mgr.14118 172.21.15.83:0/3625326745' entity='mgr.y' 2024-03-04T15:56:38.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:38 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1914249702' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-04T15:56:38.989 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:38 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ignoring --setuser ceph since I am not root 2024-03-04T15:56:38.989 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:38 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ignoring --setgroup ceph since I am not root 2024-03-04T15:56:39.384 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:39 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:39.293+0000 7f87668c5000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:56:39.384 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:39 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:39.382+0000 7f87668c5000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:56:40.074 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:39 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1914249702' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-04T15:56:40.074 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:39 smithi083 ceph-mon[24738]: mgrmap e9: y(active, since 15s) 2024-03-04T15:56:40.074 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:39 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:39.981+0000 7f87668c5000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:56:40.074 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:40 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:40.072+0000 7f87668c5000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:56:40.354 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:40 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:40.159+0000 7f87668c5000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:56:40.354 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:40 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:40.314+0000 7f87668c5000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:56:40.621 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:40 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:40.399+0000 7f87668c5000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:56:40.622 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:40 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:40.513+0000 7f87668c5000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:56:40.622 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:40 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:40.600+0000 7f87668c5000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:56:40.623 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-03-04T15:56:40.623 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 9, 2024-03-04T15:56:40.623 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "available": true, 2024-03-04T15:56:40.624 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "active_name": "y", 2024-03-04T15:56:40.624 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-04T15:56:40.624 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-03-04T15:56:40.957 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:40 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:40.770+0000 7f87668c5000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:56:40.957 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:40 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:40.858+0000 7f87668c5000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:56:41.224 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:41 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1400481363' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-04T15:56:41.227 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for the mgr to restart... 2024-03-04T15:56:41.227 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for mgr epoch 9... 2024-03-04T15:56:42.031 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:41 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:41.900+0000 7f87668c5000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:56:42.031 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:42 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:42.029+0000 7f87668c5000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:56:42.306 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:42 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:42.218+0000 7f87668c5000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:56:42.306 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:42 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:42.304+0000 7f87668c5000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:56:42.744 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:42 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:42.438+0000 7f87668c5000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:56:43.059 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:42 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:42.756+0000 7f87668c5000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:56:43.060 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:42 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:42.975+0000 7f87668c5000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:56:43.060 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:43 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:43.058+0000 7f87668c5000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:56:43.489 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:43 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:43.227+0000 7f87668c5000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:56:44.433 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:44 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:44.139+0000 7f87668c5000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:56:44.433 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:44 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:44.431+0000 7f87668c5000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:56:44.739 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:44 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:44.525+0000 7f87668c5000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:56:44.739 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:44 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:56:44.673+0000 7f87668c5000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:56:44.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:44 smithi083 ceph-mon[24738]: Active manager daemon y restarted 2024-03-04T15:56:44.999 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:44 smithi083 ceph-mon[24738]: Activating manager daemon y 2024-03-04T15:56:44.999 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:44 smithi083 ceph-mon[24738]: osdmap e3: 0 total, 0 up, 0 in 2024-03-04T15:56:45.590 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: [04/Mar/2024:15:56:45] ENGINE Bus STARTING 2024-03-04T15:56:45.590 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: [04/Mar/2024:15:56:45] ENGINE Serving on https://172.21.15.83:7150 2024-03-04T15:56:45.591 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:56:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: [04/Mar/2024:15:56:45] ENGINE Bus STARTED 2024-03-04T15:56:45.759 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-03-04T15:56:45.759 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-03-04T15:56:45.759 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "initialized": true 2024-03-04T15:56:45.759 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-03-04T15:56:45.847 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:45 smithi083 ceph-mon[24738]: mgrmap e10: y(active, starting, since 0.0551717s) 2024-03-04T15:56:45.847 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:56:45.848 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T15:56:45.848 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:56:45.848 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:56:45.848 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:56:45.848 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:45 smithi083 ceph-mon[24738]: Manager daemon y is now available 2024-03-04T15:56:45.848 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:56:45.848 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:56:45.848 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:56:45.848 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:56:45.848 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:56:46.401 INFO:teuthology.orchestra.run.smithi083.stderr:mgr epoch 9 is available 2024-03-04T15:56:46.402 INFO:teuthology.orchestra.run.smithi083.stderr:Generating a dashboard self-signed certificate... 2024-03-04T15:56:46.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:46 smithi083 ceph-mon[24738]: [04/Mar/2024:15:56:45] ENGINE Bus STARTING 2024-03-04T15:56:46.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:46 smithi083 ceph-mon[24738]: [04/Mar/2024:15:56:45] ENGINE Serving on https://172.21.15.83:7150 2024-03-04T15:56:46.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:46 smithi083 ceph-mon[24738]: [04/Mar/2024:15:56:45] ENGINE Bus STARTED 2024-03-04T15:56:46.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:46 smithi083 ceph-mon[24738]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-04T15:56:46.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:46 smithi083 ceph-mon[24738]: mgrmap e11: y(active, since 1.06305s) 2024-03-04T15:56:46.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:46 smithi083 ceph-mon[24738]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-04T15:56:46.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:46 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:56:47.161 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Self-signed certificate created 2024-03-04T15:56:47.812 INFO:teuthology.orchestra.run.smithi083.stderr:Creating initial admin user... 2024-03-04T15:56:48.129 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:48 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:56:48.129 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:48 smithi083 ceph-mon[24738]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:56:48.129 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:48 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:56:48.129 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:48 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:56:48.129 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:48 smithi083 ceph-mon[24738]: mgrmap e12: y(active, since 2s) 2024-03-04T15:56:49.280 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$huCfvo40TlSlCvoRqCWIiOuvvc3/7FUmnm/F0WVxfKNg/sLEiDQDq", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1709567809, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-04T15:56:49.835 INFO:teuthology.orchestra.run.smithi083.stderr:Fetching dashboard port number... 2024-03-04T15:56:50.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:50 smithi083 ceph-mon[24738]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:56:50.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:50 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:56:50.527 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 8443 2024-03-04T15:56:51.133 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: disabled 2024-03-04T15:56:51.140 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: inactive 2024-03-04T15:56:51.141 INFO:teuthology.orchestra.run.smithi083.stderr:firewalld.service is not enabled 2024-03-04T15:56:51.141 INFO:teuthology.orchestra.run.smithi083.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-04T15:56:51.143 INFO:teuthology.orchestra.run.smithi083.stderr:Ceph Dashboard is now available at: 2024-03-04T15:56:51.143 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-04T15:56:51.143 INFO:teuthology.orchestra.run.smithi083.stderr: URL: https://smithi083.front.sepia.ceph.com:8443/ 2024-03-04T15:56:51.143 INFO:teuthology.orchestra.run.smithi083.stderr: User: admin 2024-03-04T15:56:51.143 INFO:teuthology.orchestra.run.smithi083.stderr: Password: 2cnjs4ehmk 2024-03-04T15:56:51.143 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-04T15:56:51.143 INFO:teuthology.orchestra.run.smithi083.stderr:Enabling autotune for osd_memory_target 2024-03-04T15:56:51.413 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:51 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2179212598' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-04T15:56:52.429 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:52 smithi083 ceph-mon[24738]: mgrmap e13: y(active, since 6s) 2024-03-04T15:56:53.200 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-03-04T15:56:53.806 INFO:teuthology.orchestra.run.smithi083.stderr:You can access the Ceph CLI with: 2024-03-04T15:56:53.806 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-04T15:56:53.806 INFO:teuthology.orchestra.run.smithi083.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:56:53.806 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-04T15:56:53.806 INFO:teuthology.orchestra.run.smithi083.stderr:Please consider enabling telemetry to help improve Ceph: 2024-03-04T15:56:53.806 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-04T15:56:53.806 INFO:teuthology.orchestra.run.smithi083.stderr: ceph telemetry on 2024-03-04T15:56:53.806 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-04T15:56:53.806 INFO:teuthology.orchestra.run.smithi083.stderr:For more information see: 2024-03-04T15:56:53.807 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-04T15:56:53.807 INFO:teuthology.orchestra.run.smithi083.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-04T15:56:53.807 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-04T15:56:53.807 INFO:teuthology.orchestra.run.smithi083.stderr:Bootstrap complete. 2024-03-04T15:56:53.841 INFO:tasks.cephadm:Fetching config... 2024-03-04T15:56:53.841 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-04T15:56:53.841 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-04T15:56:53.857 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-04T15:56:53.857 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-04T15:56:53.857 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-04T15:56:53.912 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-04T15:56:53.913 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-04T15:56:53.913 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-04T15:56:53.980 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-04T15:56:53.980 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-04T15:56:53.980 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-04T15:56:54.035 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-04T15:56:54.036 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXi7O6XJJh4OL8sJZ6hxmAahkO7jy6aNf5uz1ATj5sqCR/JOGkAv3aG+P/XVxBcfWOuFJdKmC6fwx+kkTLCHKo1aeXXG+bz0AKiDDSrgsLQLLzpjQ6lMqeR/PYGvcZlm2dheKRnUSxEuA1Nzh35or6+sJLuXcxKTn/z3zFtSNRladawkHti815lHYYEDQfChqMwNPkfTjBSCTWZ8jA01ppuZXBkcV1Eqx4uVlPL+7NSY7JWNQTb6tKyEnFbnoQg0TU47YAnW/6vtEc+9YMmgT7CGIRP/tknDDQIMY75/xTQb5VkjIdiWAbTkShRpOeTzppFeX7CEOmqcmvslvHxbYojrtZf4c+7zIeNxiMIhk51MyNMZYKLnnBUzm6vSsnijQO4qn3eMZFDoBZx8dtkvRXq1jYEBbfoxllxftK4m7/lnaIQWyLIqKgnBUB3KiKZNQCPvV3mKutn1wzjPF7CbSBpTlBfUayx4po53r2afLugbrO21khhgEZppLUMTCTJiE= ceph-98ffad3c-da3f-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-04T15:56:54.113 INFO:teuthology.orchestra.run.smithi083.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXi7O6XJJh4OL8sJZ6hxmAahkO7jy6aNf5uz1ATj5sqCR/JOGkAv3aG+P/XVxBcfWOuFJdKmC6fwx+kkTLCHKo1aeXXG+bz0AKiDDSrgsLQLLzpjQ6lMqeR/PYGvcZlm2dheKRnUSxEuA1Nzh35or6+sJLuXcxKTn/z3zFtSNRladawkHti815lHYYEDQfChqMwNPkfTjBSCTWZ8jA01ppuZXBkcV1Eqx4uVlPL+7NSY7JWNQTb6tKyEnFbnoQg0TU47YAnW/6vtEc+9YMmgT7CGIRP/tknDDQIMY75/xTQb5VkjIdiWAbTkShRpOeTzppFeX7CEOmqcmvslvHxbYojrtZf4c+7zIeNxiMIhk51MyNMZYKLnnBUzm6vSsnijQO4qn3eMZFDoBZx8dtkvRXq1jYEBbfoxllxftK4m7/lnaIQWyLIqKgnBUB3KiKZNQCPvV3mKutn1wzjPF7CbSBpTlBfUayx4po53r2afLugbrO21khhgEZppLUMTCTJiE= ceph-98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:56:54.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXi7O6XJJh4OL8sJZ6hxmAahkO7jy6aNf5uz1ATj5sqCR/JOGkAv3aG+P/XVxBcfWOuFJdKmC6fwx+kkTLCHKo1aeXXG+bz0AKiDDSrgsLQLLzpjQ6lMqeR/PYGvcZlm2dheKRnUSxEuA1Nzh35or6+sJLuXcxKTn/z3zFtSNRladawkHti815lHYYEDQfChqMwNPkfTjBSCTWZ8jA01ppuZXBkcV1Eqx4uVlPL+7NSY7JWNQTb6tKyEnFbnoQg0TU47YAnW/6vtEc+9YMmgT7CGIRP/tknDDQIMY75/xTQb5VkjIdiWAbTkShRpOeTzppFeX7CEOmqcmvslvHxbYojrtZf4c+7zIeNxiMIhk51MyNMZYKLnnBUzm6vSsnijQO4qn3eMZFDoBZx8dtkvRXq1jYEBbfoxllxftK4m7/lnaIQWyLIqKgnBUB3KiKZNQCPvV3mKutn1wzjPF7CbSBpTlBfUayx4po53r2afLugbrO21khhgEZppLUMTCTJiE= ceph-98ffad3c-da3f-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-04T15:56:54.174 INFO:teuthology.orchestra.run.smithi086.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXi7O6XJJh4OL8sJZ6hxmAahkO7jy6aNf5uz1ATj5sqCR/JOGkAv3aG+P/XVxBcfWOuFJdKmC6fwx+kkTLCHKo1aeXXG+bz0AKiDDSrgsLQLLzpjQ6lMqeR/PYGvcZlm2dheKRnUSxEuA1Nzh35or6+sJLuXcxKTn/z3zFtSNRladawkHti815lHYYEDQfChqMwNPkfTjBSCTWZ8jA01ppuZXBkcV1Eqx4uVlPL+7NSY7JWNQTb6tKyEnFbnoQg0TU47YAnW/6vtEc+9YMmgT7CGIRP/tknDDQIMY75/xTQb5VkjIdiWAbTkShRpOeTzppFeX7CEOmqcmvslvHxbYojrtZf4c+7zIeNxiMIhk51MyNMZYKLnnBUzm6vSsnijQO4qn3eMZFDoBZx8dtkvRXq1jYEBbfoxllxftK4m7/lnaIQWyLIqKgnBUB3KiKZNQCPvV3mKutn1wzjPF7CbSBpTlBfUayx4po53r2afLugbrO21khhgEZppLUMTCTJiE= ceph-98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:56:54.189 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-04T15:56:54.390 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:54 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2684622451' entity='client.admin' 2024-03-04T15:56:57.903 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-04T15:56:57.904 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-04T15:56:58.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:58 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3361940804' entity='client.admin' 2024-03-04T15:56:59.592 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:59 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:56:59.592 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:59 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi083", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:56:59.592 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:59 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:56:59.592 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:59 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:56:59.592 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:56:59 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:00.478 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi086 2024-03-04T15:57:00.478 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-04T15:57:00.478 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/etc/ceph/ceph.conf 2024-03-04T15:57:00.496 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-04T15:57:00.496 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:57:00.554 INFO:tasks.cephadm:Adding host smithi086 to orchestrator... 2024-03-04T15:57:00.555 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph orch host add smithi086 2024-03-04T15:57:01.075 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:00 smithi083 ceph-mon[24738]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:57:01.075 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:00 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:01.075 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:00 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:01.075 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:00 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:01.075 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:00 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:01.075 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:00 smithi083 ceph-mon[24738]: Updating smithi083:/etc/ceph/ceph.conf 2024-03-04T15:57:01.075 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:00 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:01.075 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:00 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:02.064 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:01 smithi083 ceph-mon[24738]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:57:03.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:02 smithi083 ceph-mon[24738]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi086", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:57:03.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:02 smithi083 ceph-mon[24738]: Deploying cephadm binary to smithi086 2024-03-04T15:57:03.379 INFO:teuthology.orchestra.run.smithi083.stdout:Added host 'smithi086' with addr '172.21.15.86' 2024-03-04T15:57:04.027 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph orch host ls --format=json 2024-03-04T15:57:04.582 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:04 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:04.582 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:04 smithi083 ceph-mon[24738]: Added host smithi086 2024-03-04T15:57:04.582 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:04 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:04.582 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:04 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:04.582 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:04 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:04.582 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:04 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:04.582 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:04 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:05.937 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-04T15:57:05.938 INFO:teuthology.orchestra.run.smithi083.stdout:[{"addr": "172.21.15.83", "hostname": "smithi083", "labels": [], "status": ""}, {"addr": "172.21.15.86", "hostname": "smithi086", "labels": [], "status": ""}] 2024-03-04T15:57:05.938 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:05 smithi083 ceph-mon[24738]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:05.938 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:05 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:06.422 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-04T15:57:06.422 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph osd crush tunables default 2024-03-04T15:57:07.835 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:07 smithi083 ceph-mon[24738]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:57:07.835 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:07 smithi083 ceph-mon[24738]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:08.970 INFO:teuthology.orchestra.run.smithi083.stderr:adjusted tunables profile to default 2024-03-04T15:57:09.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:08 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:09.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:08 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:09.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:08 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi086", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:57:09.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:08 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:09.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:08 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/601053820' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-04T15:57:09.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:08 smithi083 ceph-mon[24738]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:09.532 INFO:tasks.cephadm:Adding mon.a on smithi083 2024-03-04T15:57:09.532 INFO:tasks.cephadm:Adding mon.c on smithi083 2024-03-04T15:57:09.532 INFO:tasks.cephadm:Adding mon.b on smithi086 2024-03-04T15:57:09.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph orch apply mon '3;smithi083:172.21.15.83=a;smithi083:[v2:172.21.15.83:3301,v1:172.21.15.83:6790]=c;smithi086:172.21.15.86=b' 2024-03-04T15:57:10.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:09 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/601053820' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-04T15:57:10.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:09 smithi083 ceph-mon[24738]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:57:11.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:11 smithi083 ceph-mon[24738]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:11.524 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled mon update... 2024-03-04T15:57:12.234 DEBUG:teuthology.orchestra.run.smithi083:mon.c> sudo journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mon.c.service 2024-03-04T15:57:12.236 DEBUG:teuthology.orchestra.run.smithi086:mon.b> sudo journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mon.b.service 2024-03-04T15:57:12.239 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-04T15:57:12.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:57:12.723 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:12 smithi083 ceph-mon[24738]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi083:172.21.15.83=a;smithi083:[v2:172.21.15.83:3301,v1:172.21.15.83:6790]=c;smithi086:172.21.15.86=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:57:12.723 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:12 smithi083 ceph-mon[24738]: Saving service mon spec with placement smithi083:172.21.15.83=a;smithi083:[v2:172.21.15.83:3301,v1:172.21.15.83:6790]=c;smithi086:172.21.15.86=b;count:3 2024-03-04T15:57:12.723 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:12 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:12.723 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:12 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:12.723 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:12 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:12.723 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:12 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:12.723 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:12 smithi083 ceph-mon[24738]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-04T15:57:12.724 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:12 smithi083 ceph-mon[24738]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:57:12.724 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:12 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:12.724 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:12 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:12.724 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:12 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:57:12.724 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:12 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:13.587 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:13 smithi083 ceph-mon[24738]: Deploying daemon mon.c on smithi083 2024-03-04T15:57:13.587 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:13 smithi083 ceph-mon[24738]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:14.158 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T15:57:14.158 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"98ffad3c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:55:57.037117Z","created":"2024-03-04T15:55:57.037117Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:57:14.159 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-03-04T15:57:14.834 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:14 smithi083 ceph-mon[24738]: from='client.? 172.21.15.86:0/3047119503' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:57:14.834 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:14 smithi083 systemd[1]: Starting Ceph mon.c for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T15:57:15.101 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:14 smithi083 podman[29434]: 2024-03-04 15:57:14.832790445 +0000 UTC m=+0.102111704 container create e74b0ab73a802d289c44e66156695991f2d41618264fc9971a39b124060b467e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-c, distribution-scope=public, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, ceph=True, version=8, RELEASE=HEAD, vcs-type=git, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, io.openshift.tags=base centos centos-stream) 2024-03-04T15:57:15.101 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:14 smithi083 podman[29434]: 2024-03-04 15:57:14.748769896 +0000 UTC m=+0.018091119 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-03-04T15:57:15.101 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 podman[29434]: 2024-03-04 15:57:15.077343472 +0000 UTC m=+0.346664716 container init e74b0ab73a802d289c44e66156695991f2d41618264fc9971a39b124060b467e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, GIT_CLEAN=True, com.redhat.component=centos-stream-container, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vcs-type=git, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-04T15:57:15.102 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 podman[29434]: 2024-03-04 15:57:15.080559528 +0000 UTC m=+0.349880742 container start e74b0ab73a802d289c44e66156695991f2d41618264fc9971a39b124060b467e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-c, io.openshift.expose-services=, GIT_CLEAN=True, architecture=x86_64, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, version=8, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, name=centos-stream, distribution-scope=public, vcs-type=git, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-03-04T15:57:15.102 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: set uid:gid to 167:167 (ceph:ceph) 2024-03-04T15:57:15.102 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-04T15:57:15.102 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: pidfile_write: ignore empty --pid-file 2024-03-04T15:57:15.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: load: jerasure load: lrc 2024-03-04T15:57:15.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: RocksDB version: 6.15.5 2024-03-04T15:57:15.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-04T15:57:15.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Compile date Apr 18 2022 2024-03-04T15:57:15.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: DB SUMMARY 2024-03-04T15:57:15.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: DB Session ID: TYA8OI9B6VC3YIXCVLC6 2024-03-04T15:57:15.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: CURRENT file: CURRENT 2024-03-04T15:57:15.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: IDENTITY file: IDENTITY 2024-03-04T15:57:15.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-03-04T15:57:15.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-04T15:57:15.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-04T15:57:15.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.error_if_exists: 0 2024-03-04T15:57:15.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.create_if_missing: 0 2024-03-04T15:57:15.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.paranoid_checks: 1 2024-03-04T15:57:15.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-04T15:57:15.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.env: 0x563e044c6860 2024-03-04T15:57:15.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.fs: Posix File System 2024-03-04T15:57:15.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.info_log: 0x563e05ed7dc0 2024-03-04T15:57:15.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_file_opening_threads: 16 2024-03-04T15:57:15.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.statistics: (nil) 2024-03-04T15:57:15.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.use_fsync: 0 2024-03-04T15:57:15.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_log_file_size: 0 2024-03-04T15:57:15.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-04T15:57:15.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-04T15:57:15.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.keep_log_file_num: 1000 2024-03-04T15:57:15.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.recycle_log_file_num: 0 2024-03-04T15:57:15.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.allow_fallocate: 1 2024-03-04T15:57:15.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.allow_mmap_reads: 0 2024-03-04T15:57:15.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.allow_mmap_writes: 0 2024-03-04T15:57:15.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.use_direct_reads: 0 2024-03-04T15:57:15.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-04T15:57:15.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.create_missing_column_families: 0 2024-03-04T15:57:15.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.db_log_dir: 2024-03-04T15:57:15.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-03-04T15:57:15.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-04T15:57:15.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-04T15:57:15.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-04T15:57:15.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-04T15:57:15.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-04T15:57:15.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-04T15:57:15.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.advise_random_on_open: 1 2024-03-04T15:57:15.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.db_write_buffer_size: 0 2024-03-04T15:57:15.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.write_buffer_manager: 0x563e05fc82d0 2024-03-04T15:57:15.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-04T15:57:15.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-04T15:57:15.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-04T15:57:15.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-04T15:57:15.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.rate_limiter: (nil) 2024-03-04T15:57:15.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-04T15:57:15.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.wal_recovery_mode: 2 2024-03-04T15:57:15.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.enable_thread_tracking: 0 2024-03-04T15:57:15.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.enable_pipelined_write: 0 2024-03-04T15:57:15.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.unordered_write: 0 2024-03-04T15:57:15.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-04T15:57:15.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-04T15:57:15.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-04T15:57:15.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-04T15:57:15.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.row_cache: None 2024-03-04T15:57:15.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.wal_filter: None 2024-03-04T15:57:15.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-04T15:57:15.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.allow_ingest_behind: 0 2024-03-04T15:57:15.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.preserve_deletes: 0 2024-03-04T15:57:15.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.two_write_queues: 0 2024-03-04T15:57:15.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.manual_wal_flush: 0 2024-03-04T15:57:15.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.atomic_flush: 0 2024-03-04T15:57:15.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-04T15:57:15.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-04T15:57:15.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-04T15:57:15.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.log_readahead_size: 0 2024-03-04T15:57:15.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-04T15:57:15.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.best_efforts_recovery: 0 2024-03-04T15:57:15.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-04T15:57:15.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-04T15:57:15.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.allow_data_in_errors: 0 2024-03-04T15:57:15.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.db_host_id: __hostname__ 2024-03-04T15:57:15.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_background_jobs: 2 2024-03-04T15:57:15.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_background_compactions: -1 2024-03-04T15:57:15.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_subcompactions: 1 2024-03-04T15:57:15.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-04T15:57:15.497 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-04T15:57:15.497 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-04T15:57:15.497 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_total_wal_size: 0 2024-03-04T15:57:15.497 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-04T15:57:15.497 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-04T15:57:15.497 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-04T15:57:15.497 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-04T15:57:15.497 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_open_files: -1 2024-03-04T15:57:15.497 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.bytes_per_sync: 0 2024-03-04T15:57:15.497 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-04T15:57:15.498 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-04T15:57:15.498 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compaction_readahead_size: 0 2024-03-04T15:57:15.498 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_background_flushes: -1 2024-03-04T15:57:15.498 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Compression algorithms supported: 2024-03-04T15:57:15.498 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-04T15:57:15.498 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: kZSTD supported: 0 2024-03-04T15:57:15.498 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: kXpressCompression supported: 0 2024-03-04T15:57:15.498 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: kLZ4HCCompression supported: 1 2024-03-04T15:57:15.498 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: kLZ4Compression supported: 1 2024-03-04T15:57:15.498 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: kBZip2Compression supported: 0 2024-03-04T15:57:15.498 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: kZlibCompression supported: 1 2024-03-04T15:57:15.498 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: kSnappyCompression supported: 1 2024-03-04T15:57:15.498 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-04T15:57:15.498 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-03-04T15:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-04T15:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-04T15:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.merge_operator: 2024-03-04T15:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compaction_filter: None 2024-03-04T15:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compaction_filter_factory: None 2024-03-04T15:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.sst_partitioner_factory: None 2024-03-04T15:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-04T15:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-04T15:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563e05ea3d00) 2024-03-04T15:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout: cache_index_and_filter_blocks: 1 2024-03-04T15:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-04T15:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-04T15:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout: pin_top_level_index_and_filter: 1 2024-03-04T15:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout: index_type: 0 2024-03-04T15:57:15.500 INFO:journalctl@ceph.mon.c.smithi083.stdout: data_block_index_type: 0 2024-03-04T15:57:15.500 INFO:journalctl@ceph.mon.c.smithi083.stdout: index_shortening: 1 2024-03-04T15:57:15.500 INFO:journalctl@ceph.mon.c.smithi083.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-04T15:57:15.500 INFO:journalctl@ceph.mon.c.smithi083.stdout: hash_index_allow_collision: 1 2024-03-04T15:57:15.500 INFO:journalctl@ceph.mon.c.smithi083.stdout: checksum: 1 2024-03-04T15:57:15.500 INFO:journalctl@ceph.mon.c.smithi083.stdout: no_block_cache: 0 2024-03-04T15:57:15.500 INFO:journalctl@ceph.mon.c.smithi083.stdout: block_cache: 0x563e05f0e170 2024-03-04T15:57:15.500 INFO:journalctl@ceph.mon.c.smithi083.stdout: block_cache_name: BinnedLRUCache 2024-03-04T15:57:15.500 INFO:journalctl@ceph.mon.c.smithi083.stdout: block_cache_options: 2024-03-04T15:57:15.500 INFO:journalctl@ceph.mon.c.smithi083.stdout: capacity : 536870912 2024-03-04T15:57:15.500 INFO:journalctl@ceph.mon.c.smithi083.stdout: num_shard_bits : 4 2024-03-04T15:57:15.500 INFO:journalctl@ceph.mon.c.smithi083.stdout: strict_capacity_limit : 0 2024-03-04T15:57:15.500 INFO:journalctl@ceph.mon.c.smithi083.stdout: high_pri_pool_ratio: 0.000 2024-03-04T15:57:15.500 INFO:journalctl@ceph.mon.c.smithi083.stdout: block_cache_compressed: (nil) 2024-03-04T15:57:15.501 INFO:journalctl@ceph.mon.c.smithi083.stdout: persistent_cache: (nil) 2024-03-04T15:57:15.501 INFO:journalctl@ceph.mon.c.smithi083.stdout: block_size: 4096 2024-03-04T15:57:15.501 INFO:journalctl@ceph.mon.c.smithi083.stdout: block_size_deviation: 10 2024-03-04T15:57:15.501 INFO:journalctl@ceph.mon.c.smithi083.stdout: block_restart_interval: 16 2024-03-04T15:57:15.501 INFO:journalctl@ceph.mon.c.smithi083.stdout: index_block_restart_interval: 1 2024-03-04T15:57:15.501 INFO:journalctl@ceph.mon.c.smithi083.stdout: metadata_block_size: 4096 2024-03-04T15:57:15.501 INFO:journalctl@ceph.mon.c.smithi083.stdout: partition_filters: 0 2024-03-04T15:57:15.501 INFO:journalctl@ceph.mon.c.smithi083.stdout: use_delta_encoding: 1 2024-03-04T15:57:15.501 INFO:journalctl@ceph.mon.c.smithi083.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-04T15:57:15.501 INFO:journalctl@ceph.mon.c.smithi083.stdout: whole_key_filtering: 1 2024-03-04T15:57:15.501 INFO:journalctl@ceph.mon.c.smithi083.stdout: verify_compression: 0 2024-03-04T15:57:15.501 INFO:journalctl@ceph.mon.c.smithi083.stdout: read_amp_bytes_per_bit: 0 2024-03-04T15:57:15.501 INFO:journalctl@ceph.mon.c.smithi083.stdout: format_version: 4 2024-03-04T15:57:15.501 INFO:journalctl@ceph.mon.c.smithi083.stdout: enable_index_compression: 1 2024-03-04T15:57:15.501 INFO:journalctl@ceph.mon.c.smithi083.stdout: block_align: 0 2024-03-04T15:57:15.502 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.write_buffer_size: 33554432 2024-03-04T15:57:15.502 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_write_buffer_number: 2 2024-03-04T15:57:15.502 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compression: NoCompression 2024-03-04T15:57:15.502 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.bottommost_compression: Disabled 2024-03-04T15:57:15.502 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.prefix_extractor: nullptr 2024-03-04T15:57:15.502 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-04T15:57:15.502 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.num_levels: 7 2024-03-04T15:57:15.502 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-04T15:57:15.502 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-04T15:57:15.502 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-04T15:57:15.502 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-04T15:57:15.502 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-04T15:57:15.502 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-04T15:57:15.502 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-04T15:57:15.503 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:57:15.503 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-04T15:57:15.503 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-04T15:57:15.503 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-04T15:57:15.503 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compression_opts.level: 32767 2024-03-04T15:57:15.503 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compression_opts.strategy: 0 2024-03-04T15:57:15.503 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-04T15:57:15.503 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:57:15.503 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-04T15:57:15.503 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compression_opts.enabled: false 2024-03-04T15:57:15.503 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-04T15:57:15.503 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-04T15:57:15.503 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-04T15:57:15.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.target_file_size_base: 67108864 2024-03-04T15:57:15.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-04T15:57:15.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-04T15:57:15.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-04T15:57:15.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-04T15:57:15.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-04T15:57:15.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-04T15:57:15.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-04T15:57:15.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-04T15:57:15.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-04T15:57:15.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-04T15:57:15.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-04T15:57:15.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-04T15:57:15.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-04T15:57:15.505 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.arena_block_size: 4194304 2024-03-04T15:57:15.505 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-04T15:57:15.505 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-04T15:57:15.505 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-04T15:57:15.505 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.disable_auto_compactions: 0 2024-03-04T15:57:15.505 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-04T15:57:15.505 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-04T15:57:15.505 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-04T15:57:15.505 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-04T15:57:15.505 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-04T15:57:15.505 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-04T15:57:15.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-04T15:57:15.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-04T15:57:15.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-04T15:57:15.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-04T15:57:15.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.table_properties_collectors: 2024-03-04T15:57:15.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.inplace_update_support: 0 2024-03-04T15:57:15.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-04T15:57:15.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-04T15:57:15.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-04T15:57:15.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-04T15:57:15.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.bloom_locality: 0 2024-03-04T15:57:15.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.max_successive_merges: 0 2024-03-04T15:57:15.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-04T15:57:15.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.paranoid_file_checks: 0 2024-03-04T15:57:15.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.force_consistency_checks: 1 2024-03-04T15:57:15.507 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.report_bg_io_stats: 0 2024-03-04T15:57:15.507 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.ttl: 2592000 2024-03-04T15:57:15.507 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-04T15:57:15.507 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.enable_blob_files: false 2024-03-04T15:57:15.507 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.min_blob_size: 0 2024-03-04T15:57:15.507 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.blob_file_size: 268435456 2024-03-04T15:57:15.507 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-04T15:57:15.507 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-04T15:57:15.507 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-04T15:57:15.507 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-04T15:57:15.507 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-03-04T15:57:15.507 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-03-04T15:57:15.507 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709567835105871, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-04T15:57:15.507 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-03-04T15:57:15.508 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-04T15:57:15.508 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709567835106174, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1709567835, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "934b6614-12a4-480b-9db2-ae1ee31c7c1a", "db_session_id": "TYA8OI9B6VC3YIXCVLC6"}} 2024-03-04T15:57:15.508 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-03-04T15:57:15.508 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709567835106483, "job": 1, "event": "recovery_finished"} 2024-03-04T15:57:15.508 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-04T15:57:15.508 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x563e05ef4a80 2024-03-04T15:57:15.508 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: DB pointer 0x563e05f04000 2024-03-04T15:57:15.508 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-04T15:57:15.508 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-04T15:57:15.508 INFO:journalctl@ceph.mon.c.smithi083.stdout: ** DB Stats ** 2024-03-04T15:57:15.508 INFO:journalctl@ceph.mon.c.smithi083.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:57:15.508 INFO:journalctl@ceph.mon.c.smithi083.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-04T15:57:15.508 INFO:journalctl@ceph.mon.c.smithi083.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:57:15.509 INFO:journalctl@ceph.mon.c.smithi083.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:57:15.509 INFO:journalctl@ceph.mon.c.smithi083.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-04T15:57:15.509 INFO:journalctl@ceph.mon.c.smithi083.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-04T15:57:15.509 INFO:journalctl@ceph.mon.c.smithi083.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:57:15.509 INFO:journalctl@ceph.mon.c.smithi083.stdout: 2024-03-04T15:57:15.509 INFO:journalctl@ceph.mon.c.smithi083.stdout: ** Compaction Stats [default] ** 2024-03-04T15:57:15.509 INFO:journalctl@ceph.mon.c.smithi083.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-04T15:57:15.509 INFO:journalctl@ceph.mon.c.smithi083.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T15:57:15.509 INFO:journalctl@ceph.mon.c.smithi083.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-04T15:57:15.509 INFO:journalctl@ceph.mon.c.smithi083.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-04T15:57:15.509 INFO:journalctl@ceph.mon.c.smithi083.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-04T15:57:15.509 INFO:journalctl@ceph.mon.c.smithi083.stdout: 2024-03-04T15:57:15.509 INFO:journalctl@ceph.mon.c.smithi083.stdout: ** Compaction Stats [default] ** 2024-03-04T15:57:15.510 INFO:journalctl@ceph.mon.c.smithi083.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-04T15:57:15.510 INFO:journalctl@ceph.mon.c.smithi083.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T15:57:15.510 INFO:journalctl@ceph.mon.c.smithi083.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-04T15:57:15.510 INFO:journalctl@ceph.mon.c.smithi083.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:57:15.510 INFO:journalctl@ceph.mon.c.smithi083.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-04T15:57:15.510 INFO:journalctl@ceph.mon.c.smithi083.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T15:57:15.510 INFO:journalctl@ceph.mon.c.smithi083.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T15:57:15.510 INFO:journalctl@ceph.mon.c.smithi083.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T15:57:15.510 INFO:journalctl@ceph.mon.c.smithi083.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T15:57:15.510 INFO:journalctl@ceph.mon.c.smithi083.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:57:15.510 INFO:journalctl@ceph.mon.c.smithi083.stdout: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:57:15.510 INFO:journalctl@ceph.mon.c.smithi083.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T15:57:15.510 INFO:journalctl@ceph.mon.c.smithi083.stdout: 2024-03-04T15:57:15.510 INFO:journalctl@ceph.mon.c.smithi083.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T15:57:15.510 INFO:journalctl@ceph.mon.c.smithi083.stdout: 2024-03-04T15:57:15.511 INFO:journalctl@ceph.mon.c.smithi083.stdout: ** Compaction Stats [default] ** 2024-03-04T15:57:15.511 INFO:journalctl@ceph.mon.c.smithi083.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-04T15:57:15.511 INFO:journalctl@ceph.mon.c.smithi083.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T15:57:15.511 INFO:journalctl@ceph.mon.c.smithi083.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-04T15:57:15.511 INFO:journalctl@ceph.mon.c.smithi083.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-04T15:57:15.511 INFO:journalctl@ceph.mon.c.smithi083.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-04T15:57:15.511 INFO:journalctl@ceph.mon.c.smithi083.stdout: 2024-03-04T15:57:15.511 INFO:journalctl@ceph.mon.c.smithi083.stdout: ** Compaction Stats [default] ** 2024-03-04T15:57:15.511 INFO:journalctl@ceph.mon.c.smithi083.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-04T15:57:15.511 INFO:journalctl@ceph.mon.c.smithi083.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T15:57:15.511 INFO:journalctl@ceph.mon.c.smithi083.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-04T15:57:15.511 INFO:journalctl@ceph.mon.c.smithi083.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:57:15.511 INFO:journalctl@ceph.mon.c.smithi083.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-04T15:57:15.511 INFO:journalctl@ceph.mon.c.smithi083.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T15:57:15.511 INFO:journalctl@ceph.mon.c.smithi083.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T15:57:15.512 INFO:journalctl@ceph.mon.c.smithi083.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T15:57:15.512 INFO:journalctl@ceph.mon.c.smithi083.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T15:57:15.512 INFO:journalctl@ceph.mon.c.smithi083.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:57:15.512 INFO:journalctl@ceph.mon.c.smithi083.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:57:15.512 INFO:journalctl@ceph.mon.c.smithi083.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T15:57:15.512 INFO:journalctl@ceph.mon.c.smithi083.stdout: 2024-03-04T15:57:15.512 INFO:journalctl@ceph.mon.c.smithi083.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T15:57:15.512 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-04T15:57:15.512 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: using public_addrv [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-04T15:57:15.512 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: starting mon.c rank -1 at public addrs [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] at bind addrs [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:57:15.512 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: mon.c@-1(???) e0 preinit fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:57:15.512 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: mon.c@-1(synchronizing).mds e1 new map 2024-03-04T15:57:15.513 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-04T15:57:15.513 INFO:journalctl@ceph.mon.c.smithi083.stdout: e1 2024-03-04T15:57:15.513 INFO:journalctl@ceph.mon.c.smithi083.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T15:57:15.513 INFO:journalctl@ceph.mon.c.smithi083.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T15:57:15.513 INFO:journalctl@ceph.mon.c.smithi083.stdout: legacy client fscid: -1 2024-03-04T15:57:15.513 INFO:journalctl@ceph.mon.c.smithi083.stdout: 2024-03-04T15:57:15.513 INFO:journalctl@ceph.mon.c.smithi083.stdout: No filesystems configured 2024-03-04T15:57:15.513 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-04T15:57:15.513 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-04T15:57:15.513 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-04T15:57:15.513 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-04T15:57:15.513 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-04T15:57:15.513 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-04T15:57:15.513 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-04T15:57:15.513 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:57:15.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:57:15.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:57:15.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi083:172.21.15.83=a;smithi083:[v2:172.21.15.83:3301,v1:172.21.15.83:6790]=c;smithi086:172.21.15.86=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:57:15.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: Saving service mon spec with placement smithi083:172.21.15.83=a;smithi083:[v2:172.21.15.83:3301,v1:172.21.15.83:6790]=c;smithi086:172.21.15.86=b;count:3 2024-03-04T15:57:15.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:15.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:15.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:15.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:15.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-04T15:57:15.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:57:15.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:15.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:15.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:57:15.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:15.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: Deploying daemon mon.c on smithi083 2024-03-04T15:57:15.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:15.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: from='client.? 172.21.15.86:0/3047119503' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:57:15.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-04T15:57:15.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 ceph-mon[29452]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-03-04T15:57:15.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 bash[29434]: e74b0ab73a802d289c44e66156695991f2d41618264fc9971a39b124060b467e 2024-03-04T15:57:15.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:15 smithi083 systemd[1]: Started Ceph mon.c for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T15:57:15.637 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-04T15:57:15.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:57:20.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:57:20.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: mon.a calling monitor election 2024-03-04T15:57:20.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:57:20.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:57:20.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:20.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:57:20.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: mon.c calling monitor election 2024-03-04T15:57:20.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:57:20.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:20.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:57:20.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:57:20.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T15:57:20.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: monmap e2: 2 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],c=[v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0]} 2024-03-04T15:57:20.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: fsmap 2024-03-04T15:57:20.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:57:20.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: mgrmap e13: y(active, since 35s) 2024-03-04T15:57:20.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: overall HEALTH_OK 2024-03-04T15:57:20.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:20.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:57:20.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:20.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:57:20.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: mon.a calling monitor election 2024-03-04T15:57:20.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:57:20.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:57:20.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:20.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:57:20.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: mon.c calling monitor election 2024-03-04T15:57:20.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:57:20.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:20.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:57:20.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:57:20.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T15:57:20.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: monmap e2: 2 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],c=[v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0]} 2024-03-04T15:57:20.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: fsmap 2024-03-04T15:57:20.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:57:20.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: mgrmap e13: y(active, since 35s) 2024-03-04T15:57:20.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: overall HEALTH_OK 2024-03-04T15:57:20.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:20.493 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:57:20.493 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:20 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:20.605 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T15:57:20.605 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":2,"fsid":"98ffad3c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:57:15.128214Z","created":"2024-03-04T15:55:57.037117Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-04T15:57:20.606 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 2 2024-03-04T15:57:21.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:21 smithi083 ceph-mon[24738]: Deploying daemon mon.b on smithi086 2024-03-04T15:57:21.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:21 smithi083 ceph-mon[24738]: from='client.? 172.21.15.86:0/4010438937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:57:21.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:21 smithi083 ceph-mon[24738]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:21.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:21 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:57:21.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:21 smithi083 ceph-mon[29452]: Deploying daemon mon.b on smithi086 2024-03-04T15:57:21.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:21 smithi083 ceph-mon[29452]: from='client.? 172.21.15.86:0/4010438937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:57:21.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:21 smithi083 ceph-mon[29452]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:21.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:21 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:57:23.513 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-04T15:57:23.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:57:24.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:23 smithi083 ceph-mon[29452]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:24.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:23 smithi083 ceph-mon[24738]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:30.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:57:30.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: mon.a calling monitor election 2024-03-04T15:57:30.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: mon.c calling monitor election 2024-03-04T15:57:30.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:57:30.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:57:30.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:57:30.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:30.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:57:30.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:57:30.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:30.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:57:30.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:57:30.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T15:57:30.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: monmap e3: 3 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],b=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],c=[v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0]} 2024-03-04T15:57:30.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: fsmap 2024-03-04T15:57:30.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:57:30.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: mgrmap e13: y(active, since 45s) 2024-03-04T15:57:30.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-04T15:57:30.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T15:57:30.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T15:57:30.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: mon.b (rank 2) addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] is down (out of quorum) 2024-03-04T15:57:30.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:30.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:30.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:30.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:30.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:57:30.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: mon.a calling monitor election 2024-03-04T15:57:30.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: mon.c calling monitor election 2024-03-04T15:57:30.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:57:30.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:57:30.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:57:30.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:30.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:57:30.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:57:30.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:30.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:57:30.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:57:30.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T15:57:30.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: monmap e3: 3 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],b=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],c=[v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0]} 2024-03-04T15:57:30.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: fsmap 2024-03-04T15:57:30.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:57:30.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: mgrmap e13: y(active, since 45s) 2024-03-04T15:57:30.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-04T15:57:30.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T15:57:30.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T15:57:30.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: mon.b (rank 2) addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] is down (out of quorum) 2024-03-04T15:57:30.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:30.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:30.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:30.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:30 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:31.553 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:31 smithi083 ceph-mon[24738]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:31.553 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:31 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:57:31.554 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:31 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:31.554 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:31 smithi083 ceph-mon[29452]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:31.554 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:31 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:57:31.554 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:31 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:31.879 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T15:57:31.879 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":3,"fsid":"98ffad3c-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:57:25.199635Z","created":"2024-03-04T15:55:57.037117Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-04T15:57:31.881 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 3 2024-03-04T15:57:32.436 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:32 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:32.458 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-04T15:57:32.459 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph config generate-minimal-conf 2024-03-04T15:57:32.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:32 smithi086 ceph-mon[25584]: mon.b calling monitor election 2024-03-04T15:57:32.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:32 smithi086 ceph-mon[25584]: mon.b calling monitor election 2024-03-04T15:57:32.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:32 smithi086 ceph-mon[25584]: mon.c calling monitor election 2024-03-04T15:57:32.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:32 smithi086 ceph-mon[25584]: mon.a calling monitor election 2024-03-04T15:57:32.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:32 smithi086 ceph-mon[25584]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T15:57:32.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:32 smithi086 ceph-mon[25584]: monmap e3: 3 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],b=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],c=[v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0]} 2024-03-04T15:57:32.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:32 smithi086 ceph-mon[25584]: fsmap 2024-03-04T15:57:32.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:32 smithi086 ceph-mon[25584]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:57:32.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:32 smithi086 ceph-mon[25584]: mgrmap e13: y(active, since 47s) 2024-03-04T15:57:32.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:32 smithi086 ceph-mon[25584]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-04T15:57:32.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:32 smithi086 ceph-mon[25584]: Cluster is now healthy 2024-03-04T15:57:32.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:32 smithi086 ceph-mon[25584]: overall HEALTH_OK 2024-03-04T15:57:32.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[29452]: mon.b calling monitor election 2024-03-04T15:57:32.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[29452]: mon.b calling monitor election 2024-03-04T15:57:32.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[29452]: mon.c calling monitor election 2024-03-04T15:57:32.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[29452]: mon.a calling monitor election 2024-03-04T15:57:32.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[29452]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T15:57:32.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[29452]: monmap e3: 3 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],b=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],c=[v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0]} 2024-03-04T15:57:32.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[29452]: fsmap 2024-03-04T15:57:32.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[29452]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:57:32.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[29452]: mgrmap e13: y(active, since 47s) 2024-03-04T15:57:32.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[29452]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-04T15:57:32.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[29452]: Cluster is now healthy 2024-03-04T15:57:32.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[29452]: overall HEALTH_OK 2024-03-04T15:57:32.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[24738]: mon.b calling monitor election 2024-03-04T15:57:32.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[24738]: mon.b calling monitor election 2024-03-04T15:57:32.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[24738]: mon.c calling monitor election 2024-03-04T15:57:32.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[24738]: mon.a calling monitor election 2024-03-04T15:57:32.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[24738]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T15:57:32.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[24738]: monmap e3: 3 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],b=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],c=[v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0]} 2024-03-04T15:57:32.742 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[24738]: fsmap 2024-03-04T15:57:32.742 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[24738]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:57:32.742 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[24738]: mgrmap e13: y(active, since 47s) 2024-03-04T15:57:32.742 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[24738]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-04T15:57:32.742 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[24738]: Cluster is now healthy 2024-03-04T15:57:32.742 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:32 smithi083 ceph-mon[24738]: overall HEALTH_OK 2024-03-04T15:57:33.852 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:33 smithi083 ceph-mon[24738]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:33.853 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:33 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:33.853 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:33 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:57:33.853 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:33 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:57:33.853 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:33 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:33.853 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:33 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:57:33.853 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:33 smithi083 ceph-mon[29452]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:33.853 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:33 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:33.853 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:33 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:57:33.853 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:33 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:57:33.853 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:33 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:33.854 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:33 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:57:33.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:33 smithi086 ceph-mon[25584]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:33.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:33 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:33.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:33 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:57:33.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:33 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:57:33.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:33 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:33.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:33 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:57:34.473 INFO:teuthology.orchestra.run.smithi083.stdout:# minimal ceph.conf for 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:57:34.473 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2024-03-04T15:57:34.473 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T15:57:34.473 INFO:teuthology.orchestra.run.smithi083.stdout: mon_host = [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-04T15:57:34.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:34 smithi083 ceph-mon[24738]: Reconfiguring mon.c (monmap changed)... 2024-03-04T15:57:34.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:34 smithi083 ceph-mon[24738]: Reconfiguring daemon mon.c on smithi083 2024-03-04T15:57:34.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:34 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2790863979' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:34.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:34 smithi083 ceph-mon[29452]: Reconfiguring mon.c (monmap changed)... 2024-03-04T15:57:34.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:34 smithi083 ceph-mon[29452]: Reconfiguring daemon mon.c on smithi083 2024-03-04T15:57:34.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:34 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/2790863979' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:34.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:34 smithi086 ceph-mon[25584]: Reconfiguring mon.c (monmap changed)... 2024-03-04T15:57:34.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:34 smithi086 ceph-mon[25584]: Reconfiguring daemon mon.c on smithi083 2024-03-04T15:57:34.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:34 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/2790863979' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:35.881 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-04T15:57:35.881 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-04T15:57:35.881 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.conf 2024-03-04T15:57:35.910 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-04T15:57:35.910 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:57:35.974 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:35 smithi083 ceph-mon[29452]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:35.974 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:35 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:35.974 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:35 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:35.974 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:35 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:57:35.974 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:35 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:57:35.974 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:35 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:35.975 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:35 smithi083 ceph-mon[24738]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:35.975 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:35 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:35.975 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:35 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:35.975 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:35 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:57:35.975 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:35 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:57:35.975 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:35 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:35.977 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-04T15:57:35.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.conf 2024-03-04T15:57:36.045 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-04T15:57:36.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:57:36.049 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:35 smithi086 ceph-mon[25584]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:36.049 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:35 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:36.049 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:35 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:36.049 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:35 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:57:36.050 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:35 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:57:36.050 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:35 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:36.073 INFO:tasks.cephadm:Adding mgr.y on smithi083 2024-03-04T15:57:36.073 INFO:tasks.cephadm:Adding mgr.x on smithi086 2024-03-04T15:57:36.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph orch apply mgr '2;smithi083=y;smithi086=x' 2024-03-04T15:57:36.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:36 smithi086 ceph-mon[25584]: Reconfiguring mon.b (monmap changed)... 2024-03-04T15:57:36.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:36 smithi086 ceph-mon[25584]: Reconfiguring daemon mon.b on smithi086 2024-03-04T15:57:36.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:36 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:36.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:36 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:36.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:36 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:36.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:36 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:36.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:36 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:36.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:36 smithi083 ceph-mon[29452]: Reconfiguring mon.b (monmap changed)... 2024-03-04T15:57:36.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:36 smithi083 ceph-mon[29452]: Reconfiguring daemon mon.b on smithi086 2024-03-04T15:57:36.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:36 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:36.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:36 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:36.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:36 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:36.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:36 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:36.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:36 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:36.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:36 smithi083 ceph-mon[24738]: Reconfiguring mon.b (monmap changed)... 2024-03-04T15:57:36.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:36 smithi083 ceph-mon[24738]: Reconfiguring daemon mon.b on smithi086 2024-03-04T15:57:36.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:36 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:36.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:36 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:36.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:36 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:36.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:36 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:36.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:36 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:37.932 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:37 smithi086 ceph-mon[25584]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:38.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:37 smithi083 ceph-mon[24738]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:38.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:37 smithi083 ceph-mon[29452]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:38.308 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled mgr update... 2024-03-04T15:57:38.837 DEBUG:teuthology.orchestra.run.smithi086:mgr.x> sudo journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mgr.x.service 2024-03-04T15:57:38.839 INFO:tasks.cephadm:Deploying OSDs... 2024-03-04T15:57:38.839 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-04T15:57:38.839 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:57:38.860 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T15:57:38.860 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_1 2024-03-04T15:57:38.918 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T15:57:38.918 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:57:38.918 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-04T15:57:38.918 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:57:38.918 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:57:38.918 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-04 15:57:14.573923319 +0000 2024-03-04T15:57:38.918 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-04 15:54:54.632332796 +0000 2024-03-04T15:57:38.918 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-04 15:54:54.632332796 +0000 2024-03-04T15:57:38.918 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-04 15:54:54.632332796 +0000 2024-03-04T15:57:38.919 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T15:57:38.989 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-04T15:57:38.989 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-04T15:57:38.989 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000240639 s, 2.1 MB/s 2024-03-04T15:57:38.990 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T15:57:39.049 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_2 2024-03-04T15:57:39.106 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T15:57:39.107 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:57:39.107 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-04T15:57:39.107 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:57:39.107 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:57:39.107 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-04 15:57:14.573923319 +0000 2024-03-04T15:57:39.107 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-04 15:54:54.961325403 +0000 2024-03-04T15:57:39.107 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-04 15:54:54.961325403 +0000 2024-03-04T15:57:39.107 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-04 15:54:54.961325403 +0000 2024-03-04T15:57:39.107 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T15:57:39.176 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-04T15:57:39.177 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-04T15:57:39.177 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000222196 s, 2.3 MB/s 2024-03-04T15:57:39.178 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T15:57:39.236 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_3 2024-03-04T15:57:39.293 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T15:57:39.293 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:57:39.293 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-04T15:57:39.293 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:57:39.293 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:57:39.293 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-04 15:57:14.573923319 +0000 2024-03-04T15:57:39.293 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-04 15:54:55.287318078 +0000 2024-03-04T15:57:39.293 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-04 15:54:55.287318078 +0000 2024-03-04T15:57:39.293 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-04 15:54:55.287318078 +0000 2024-03-04T15:57:39.294 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T15:57:39.362 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-04T15:57:39.362 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-04T15:57:39.363 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000242956 s, 2.1 MB/s 2024-03-04T15:57:39.364 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T15:57:39.421 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_4 2024-03-04T15:57:39.477 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T15:57:39.477 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:57:39.477 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-04T15:57:39.477 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:57:39.478 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:57:39.478 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-04 15:57:14.574923297 +0000 2024-03-04T15:57:39.478 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-04 15:54:55.622310550 +0000 2024-03-04T15:57:39.478 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-04 15:54:55.622310550 +0000 2024-03-04T15:57:39.478 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-04 15:54:55.622310550 +0000 2024-03-04T15:57:39.478 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T15:57:39.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:39 smithi086 ceph-mon[25584]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi083=y;smithi086=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:57:39.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:39 smithi086 ceph-mon[25584]: Saving service mgr spec with placement smithi083=y;smithi086=x;count:2 2024-03-04T15:57:39.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:39 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:39.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:39 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:39.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:39 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:39.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:39 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:39.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:39 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:39.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:39 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:57:39.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:39 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T15:57:39.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:39 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:57:39.487 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:39 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:39.487 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:39 smithi086 ceph-mon[25584]: Deploying daemon mgr.x on smithi086 2024-03-04T15:57:39.487 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:39 smithi086 ceph-mon[25584]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:39.544 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-04T15:57:39.545 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-04T15:57:39.545 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000180755 s, 2.8 MB/s 2024-03-04T15:57:39.546 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T15:57:39.602 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-04T15:57:39.603 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:57:39.618 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T15:57:39.619 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2024-03-04T15:57:39.673 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T15:57:39.673 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:57:39.674 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-04T15:57:39.674 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:57:39.674 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:57:39.674 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-04 15:57:23.787876138 +0000 2024-03-04T15:57:39.674 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-04 15:54:54.647947335 +0000 2024-03-04T15:57:39.674 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-04 15:54:54.647947335 +0000 2024-03-04T15:57:39.674 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-04 15:54:54.647947335 +0000 2024-03-04T15:57:39.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T15:57:39.739 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-04T15:57:39.739 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-04T15:57:39.739 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000208908 s, 2.5 MB/s 2024-03-04T15:57:39.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[29452]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi083=y;smithi086=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:57:39.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[29452]: Saving service mgr spec with placement smithi083=y;smithi086=x;count:2 2024-03-04T15:57:39.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:39.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:39.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:39.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:39.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:39.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:57:39.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T15:57:39.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:57:39.744 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:39.744 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[29452]: Deploying daemon mgr.x on smithi086 2024-03-04T15:57:39.744 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[29452]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:39.745 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[24738]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi083=y;smithi086=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:57:39.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[24738]: Saving service mgr spec with placement smithi083=y;smithi086=x;count:2 2024-03-04T15:57:39.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:39.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:39.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:39.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:39.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:39.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:57:39.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T15:57:39.747 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:57:39.747 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:39.747 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[24738]: Deploying daemon mgr.x on smithi086 2024-03-04T15:57:39.747 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:39 smithi083 ceph-mon[24738]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:39.748 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T15:57:39.803 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2024-03-04T15:57:39.863 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T15:57:39.863 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:57:39.863 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-04T15:57:39.863 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:57:39.864 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:57:39.864 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-04 15:57:23.787876138 +0000 2024-03-04T15:57:39.864 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-04 15:54:54.988940129 +0000 2024-03-04T15:57:39.864 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-04 15:54:54.988940129 +0000 2024-03-04T15:57:39.864 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-04 15:54:54.988940129 +0000 2024-03-04T15:57:39.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T15:57:40.031 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-04T15:57:40.031 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-04T15:57:40.031 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000209055 s, 2.4 MB/s 2024-03-04T15:57:40.032 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T15:57:40.054 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2024-03-04T15:57:40.117 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T15:57:40.117 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:57:40.117 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-04T15:57:40.117 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:57:40.118 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:57:40.118 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-04 15:57:39.988531559 +0000 2024-03-04T15:57:40.118 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-04 15:54:55.321933093 +0000 2024-03-04T15:57:40.118 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-04 15:54:55.321933093 +0000 2024-03-04T15:57:40.118 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-04 15:54:55.321933093 +0000 2024-03-04T15:57:40.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T15:57:40.209 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-04T15:57:40.210 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-04T15:57:40.210 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000172754 s, 3.0 MB/s 2024-03-04T15:57:40.210 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T15:57:40.269 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2024-03-04T15:57:40.326 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T15:57:40.326 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:57:40.326 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-04T15:57:40.326 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:57:40.326 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:57:40.327 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-04 15:57:40.166527676 +0000 2024-03-04T15:57:40.327 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-04 15:54:55.663925866 +0000 2024-03-04T15:57:40.327 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-04 15:54:55.663925866 +0000 2024-03-04T15:57:40.327 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-04 15:54:55.663925866 +0000 2024-03-04T15:57:40.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T15:57:40.393 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:40 smithi086 systemd[1]: Starting Ceph mgr.x for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T15:57:40.396 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-04T15:57:40.396 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-04T15:57:40.396 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000210311 s, 2.4 MB/s 2024-03-04T15:57:40.397 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T15:57:40.453 INFO:tasks.cephadm:Deploying osd.0 on smithi083 with /dev/vg_nvme/lv_4... 2024-03-04T15:57:40.453 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-04T15:57:40.661 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:40 smithi086 podman[26614]: 2024-03-04 15:57:40.340221726 +0000 UTC m=+0.019280982 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-03-04T15:57:40.661 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:40 smithi086 podman[26614]: 2024-03-04 15:57:40.458688718 +0000 UTC m=+0.137747962 container create 799a7f92600598f1844b1642991338a35b4c3a4efb32e443c61b625f82a26abf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, ceph=True, release=754, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.openshift.tags=base centos centos-stream, distribution-scope=public, architecture=x86_64, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, vendor=Red Hat, Inc.) 2024-03-04T15:57:40.662 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:40 smithi086 podman[26614]: 2024-03-04 15:57:40.660834623 +0000 UTC m=+0.339893867 container init 799a7f92600598f1844b1642991338a35b4c3a4efb32e443c61b625f82a26abf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, distribution-scope=public, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-04T15:57:40.914 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:40 smithi086 podman[26614]: 2024-03-04 15:57:40.663590789 +0000 UTC m=+0.342650033 container start 799a7f92600598f1844b1642991338a35b4c3a4efb32e443c61b625f82a26abf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-04T15:57:40.914 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:40 smithi086 ceph-mgr[26681]: -- 172.21.15.86:0/2962255982 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56112c62c340 con 0x56112c604800 2024-03-04T15:57:40.914 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:40 smithi086 bash[26614]: 799a7f92600598f1844b1642991338a35b4c3a4efb32e443c61b625f82a26abf 2024-03-04T15:57:40.915 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:40 smithi086 systemd[1]: Started Ceph mgr.x for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T15:57:41.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:41 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:41.177+0000 7faf22a9c000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:57:41.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:41 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:41.269+0000 7faf22a9c000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:57:41.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:41 smithi083 ceph-mon[29452]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:41.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:41 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:41.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:41 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:41.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:41 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:41.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:41 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:41.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:41 smithi083 ceph-mon[24738]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:41.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:41 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:41.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:41 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:41.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:41 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:41.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:41 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:42.211 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:41 smithi086 ceph-mon[25584]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:42.211 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:41 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:42.211 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:41 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:42.211 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:41 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:42.211 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:41 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:42.212 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:41 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:41.894+0000 7faf22a9c000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:57:42.212 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:41 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:41.981+0000 7faf22a9c000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:57:42.212 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:42 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:42.062+0000 7faf22a9c000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:57:42.480 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-04T15:57:42.480 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-04T15:57:42.480 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-03-04T15:57:42.480 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-03-04T15:57:42.480 INFO:teuthology.orchestra.run.smithi083.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0127187 s, 824 MB/s 2024-03-04T15:57:42.480 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 2024-03-04T15:57:42.480 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-03-04T15:57:42.481 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:42 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:42.209+0000 7faf22a9c000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:57:42.481 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:42 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:42.289+0000 7faf22a9c000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:57:42.481 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:42 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:42.396+0000 7faf22a9c000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:57:42.481 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:42 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:42.478+0000 7faf22a9c000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:57:42.735 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:42 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:42.639+0000 7faf22a9c000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:57:42.735 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:42 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:42.723+0000 7faf22a9c000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:57:43.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:42 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:43.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:42 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:43.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:42 smithi086 ceph-mon[25584]: Reconfiguring mgr.y (unknown last config time)... 2024-03-04T15:57:43.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:42 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:57:43.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:42 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:57:43.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:42 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:43.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:42 smithi086 ceph-mon[25584]: Reconfiguring daemon mgr.y on smithi083 2024-03-04T15:57:43.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:42 smithi086 ceph-mon[25584]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:43.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:42 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:43.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:42 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:43.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:42 smithi083 ceph-mon[24738]: Reconfiguring mgr.y (unknown last config time)... 2024-03-04T15:57:43.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:42 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:57:43.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:42 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:57:43.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:42 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:43.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:42 smithi083 ceph-mon[24738]: Reconfiguring daemon mgr.y on smithi083 2024-03-04T15:57:43.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:42 smithi083 ceph-mon[24738]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:43.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:42 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:43.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:42 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:43.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:42 smithi083 ceph-mon[29452]: Reconfiguring mgr.y (unknown last config time)... 2024-03-04T15:57:43.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:42 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:57:43.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:42 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:57:43.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:42 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:43.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:42 smithi083 ceph-mon[29452]: Reconfiguring daemon mgr.y on smithi083 2024-03-04T15:57:43.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:42 smithi083 ceph-mon[29452]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:43.803 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi083:vg_nvme/lv_4 2024-03-04T15:57:44.061 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:43 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:43.742+0000 7faf22a9c000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:57:44.062 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:43 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:43.870+0000 7faf22a9c000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:57:44.062 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:44.059+0000 7faf22a9c000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:57:44.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:44.145+0000 7faf22a9c000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:57:44.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:44.281+0000 7faf22a9c000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:57:44.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:44 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:44.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:44 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:44.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:44 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:44.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:44 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:44.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:44 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:44.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:44 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:44.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:44 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:44.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:44 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:44.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:44 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:44.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:44 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:44.835 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:44.616+0000 7faf22a9c000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:57:44.836 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:44 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:44.836 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:44 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:44.836 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:44 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:44.836 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:44 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:44.836 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:44 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:45.089 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:44.834+0000 7faf22a9c000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:57:45.089 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:44.917+0000 7faf22a9c000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:57:45.089 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:45 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:45.087+0000 7faf22a9c000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:57:45.723 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:45 smithi083 ceph-mon[24738]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:45.724 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:57:45.724 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:57:45.724 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:45.724 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:45 smithi083 ceph-mon[29452]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:45.725 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:45 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:57:45.725 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:45 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:57:45.725 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:45 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:45.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:45 smithi086 ceph-mon[25584]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:45.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:45 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:57:45.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:45 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:57:45.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:45 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:46.309 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:46 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:46.019+0000 7faf22a9c000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:57:46.309 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:46 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:46.307+0000 7faf22a9c000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:57:46.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:46 smithi086 ceph-mon[25584]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:57:46.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:46 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:57:46.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:46 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:57:46.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:46 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:46.735 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:46 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:46.401+0000 7faf22a9c000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:57:46.735 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 15:57:46 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T15:57:46.550+0000 7faf22a9c000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:57:46.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:46 smithi083 ceph-mon[24738]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:57:46.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:46 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:57:46.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:46 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:57:46.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:46 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:46.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:46 smithi083 ceph-mon[29452]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:57:46.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:46 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:57:46.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:46 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:57:46.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:46 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:47.563 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:47 smithi083 ceph-mon[24738]: Standby manager daemon x started 2024-03-04T15:57:47.563 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:47 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/937287671' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T15:57:47.564 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:47 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/937287671' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:57:47.564 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:47 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/937287671' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T15:57:47.564 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:47 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/937287671' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:57:47.564 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:47 smithi083 ceph-mon[24738]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:47.564 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:47 smithi083 ceph-mon[29452]: Standby manager daemon x started 2024-03-04T15:57:47.564 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:47 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/937287671' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T15:57:47.564 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:47 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/937287671' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:57:47.564 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:47 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/937287671' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T15:57:47.564 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:47 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/937287671' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:57:47.564 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:47 smithi083 ceph-mon[29452]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:47.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:47 smithi086 ceph-mon[25584]: Standby manager daemon x started 2024-03-04T15:57:47.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:47 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/937287671' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T15:57:47.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:47 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/937287671' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:57:47.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:47 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/937287671' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T15:57:47.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:47 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/937287671' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:57:47.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:47 smithi086 ceph-mon[25584]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:48.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:48 smithi086 ceph-mon[25584]: mgrmap e14: y(active, since 62s), standbys: x 2024-03-04T15:57:48.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:48 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T15:57:48.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:48 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1427446024' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87dbf1ea-c0a9-4218-8659-1120adebf7e9"}]: dispatch 2024-03-04T15:57:48.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:48 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1427446024' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "87dbf1ea-c0a9-4218-8659-1120adebf7e9"}]': finished 2024-03-04T15:57:48.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:48 smithi086 ceph-mon[25584]: osdmap e5: 1 total, 0 up, 1 in 2024-03-04T15:57:48.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:48 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:57:48.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:48 smithi083 ceph-mon[24738]: mgrmap e14: y(active, since 62s), standbys: x 2024-03-04T15:57:48.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:48 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T15:57:48.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:48 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1427446024' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87dbf1ea-c0a9-4218-8659-1120adebf7e9"}]: dispatch 2024-03-04T15:57:48.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:48 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1427446024' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "87dbf1ea-c0a9-4218-8659-1120adebf7e9"}]': finished 2024-03-04T15:57:48.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:48 smithi083 ceph-mon[24738]: osdmap e5: 1 total, 0 up, 1 in 2024-03-04T15:57:48.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:48 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:57:48.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:48 smithi083 ceph-mon[29452]: mgrmap e14: y(active, since 62s), standbys: x 2024-03-04T15:57:48.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:48 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T15:57:48.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:48 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1427446024' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87dbf1ea-c0a9-4218-8659-1120adebf7e9"}]: dispatch 2024-03-04T15:57:48.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:48 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1427446024' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "87dbf1ea-c0a9-4218-8659-1120adebf7e9"}]': finished 2024-03-04T15:57:48.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:48 smithi083 ceph-mon[29452]: osdmap e5: 1 total, 0 up, 1 in 2024-03-04T15:57:48.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:48 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:57:49.988 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:49 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3001738156' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:57:49.988 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:49 smithi086 ceph-mon[25584]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:49.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:49 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3001738156' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:57:49.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:49 smithi083 ceph-mon[24738]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:49.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:49 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3001738156' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:57:49.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:49 smithi083 ceph-mon[29452]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:51.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:51 smithi083 ceph-mon[24738]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:51.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:51 smithi083 ceph-mon[29452]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:51.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:51 smithi086 ceph-mon[25584]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:53.906 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:53 smithi083 ceph-mon[24738]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:53.906 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:53 smithi083 ceph-mon[29452]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:54.188 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:53 smithi086 ceph-mon[25584]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:54.918 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:54 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T15:57:54.918 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:54 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:54.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:54 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T15:57:54.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:54 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:55.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:54 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T15:57:55.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:54 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:56.147 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:55 smithi083 ceph-mon[24738]: Deploying daemon osd.0 on smithi083 2024-03-04T15:57:56.147 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:55 smithi083 ceph-mon[24738]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:56.148 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:55 smithi083 ceph-mon[29452]: Deploying daemon osd.0 on smithi083 2024-03-04T15:57:56.148 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:55 smithi083 ceph-mon[29452]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:56.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:55 smithi086 ceph-mon[25584]: Deploying daemon osd.0 on smithi083 2024-03-04T15:57:56.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:55 smithi086 ceph-mon[25584]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:57.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:57 smithi083 ceph-mon[24738]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:57.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:57 smithi083 ceph-mon[29452]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:58.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:57 smithi086 ceph-mon[25584]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:59.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:59 smithi083 ceph-mon[29452]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:59.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:59 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:59.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:59 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:59.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:59 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:59.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:57:59 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:57:59.891 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:59 smithi083 ceph-mon[24738]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:57:59.891 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:59 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:57:59.892 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:59 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:59.892 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:59 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:57:59.892 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:57:59 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:00.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:59 smithi086 ceph-mon[25584]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:00.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:59 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:00.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:59 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:00.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:59 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:00.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:57:59 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:01.142 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:01 smithi083 ceph-mon[29452]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:01.143 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:01 smithi083 ceph-mon[24738]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:01.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:01 smithi086 ceph-mon[25584]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:02.152 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 0 on host 'smithi083' 2024-03-04T15:58:02.799 DEBUG:teuthology.orchestra.run.smithi083:osd.0> sudo journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.0.service 2024-03-04T15:58:02.802 INFO:tasks.cephadm:Deploying osd.1 on smithi083 with /dev/vg_nvme/lv_3... 2024-03-04T15:58:02.802 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-04T15:58:03.386 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:03 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:03.386 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:03 smithi083 ceph-mon[24738]: from='osd.0 [v2:172.21.15.83:6802/3013656475,v1:172.21.15.83:6803/3013656475]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T15:58:03.386 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:03 smithi083 ceph-mon[24738]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:03.386 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:03 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:03.387 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:03 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:03.387 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:03 smithi083 ceph-mon[29452]: from='osd.0 [v2:172.21.15.83:6802/3013656475,v1:172.21.15.83:6803/3013656475]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T15:58:03.387 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:03 smithi083 ceph-mon[29452]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:03.387 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:03 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:03.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:03 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:03.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:03 smithi086 ceph-mon[25584]: from='osd.0 [v2:172.21.15.83:6802/3013656475,v1:172.21.15.83:6803/3013656475]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T15:58:03.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:03 smithi086 ceph-mon[25584]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:03.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:03 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:04.378 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:04 smithi083 ceph-mon[24738]: from='osd.0 [v2:172.21.15.83:6802/3013656475,v1:172.21.15.83:6803/3013656475]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T15:58:04.378 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:04 smithi083 ceph-mon[24738]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T15:58:04.378 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:04 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:58:04.378 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:04 smithi083 ceph-mon[24738]: from='osd.0 [v2:172.21.15.83:6802/3013656475,v1:172.21.15.83:6803/3013656475]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T15:58:04.378 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:04 smithi083 ceph-mon[29452]: from='osd.0 [v2:172.21.15.83:6802/3013656475,v1:172.21.15.83:6803/3013656475]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T15:58:04.379 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:04 smithi083 ceph-mon[29452]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T15:58:04.379 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:04 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:58:04.379 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:04 smithi083 ceph-mon[29452]: from='osd.0 [v2:172.21.15.83:6802/3013656475,v1:172.21.15.83:6803/3013656475]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T15:58:04.379 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 15:58:04 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0[32297]: 2024-03-04T15:58:04.158+0000 7f4381b1d700 -1 osd.0 0 waiting for initial osdmap 2024-03-04T15:58:04.379 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 15:58:04 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0[32297]: 2024-03-04T15:58:04.164+0000 7f437bcb3700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T15:58:04.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:04 smithi086 ceph-mon[25584]: from='osd.0 [v2:172.21.15.83:6802/3013656475,v1:172.21.15.83:6803/3013656475]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T15:58:04.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:04 smithi086 ceph-mon[25584]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T15:58:04.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:04 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:58:04.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:04 smithi086 ceph-mon[25584]: from='osd.0 [v2:172.21.15.83:6802/3013656475,v1:172.21.15.83:6803/3013656475]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T15:58:05.387 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-04T15:58:05.387 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-04T15:58:05.387 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-03-04T15:58:05.387 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-03-04T15:58:05.387 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0143488 s, 731 MB/s 2024-03-04T15:58:05.387 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-03-04T15:58:05.438 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:05 smithi083 ceph-mon[24738]: from='osd.0 [v2:172.21.15.83:6802/3013656475,v1:172.21.15.83:6803/3013656475]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-04T15:58:05.439 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:05 smithi083 ceph-mon[24738]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T15:58:05.439 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:05 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:58:05.439 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:05 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:58:05.439 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:05 smithi083 ceph-mon[24738]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:05.439 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:05 smithi083 ceph-mon[29452]: from='osd.0 [v2:172.21.15.83:6802/3013656475,v1:172.21.15.83:6803/3013656475]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-04T15:58:05.439 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:05 smithi083 ceph-mon[29452]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T15:58:05.439 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:05 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:58:05.439 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:05 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:58:05.439 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:05 smithi083 ceph-mon[29452]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:05.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:05 smithi086 ceph-mon[25584]: from='osd.0 [v2:172.21.15.83:6802/3013656475,v1:172.21.15.83:6803/3013656475]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-04T15:58:05.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:05 smithi086 ceph-mon[25584]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T15:58:05.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:05 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:58:05.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:05 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:58:05.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:05 smithi086 ceph-mon[25584]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:06.171 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi083:vg_nvme/lv_3 2024-03-04T15:58:06.422 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:06 smithi083 ceph-mon[29452]: purged_snaps scrub starts 2024-03-04T15:58:06.422 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:06 smithi083 ceph-mon[29452]: purged_snaps scrub ok 2024-03-04T15:58:06.422 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:06 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:58:06.422 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:06 smithi083 ceph-mon[29452]: osd.0 [v2:172.21.15.83:6802/3013656475,v1:172.21.15.83:6803/3013656475] boot 2024-03-04T15:58:06.422 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:06 smithi083 ceph-mon[29452]: osdmap e8: 1 total, 1 up, 1 in 2024-03-04T15:58:06.422 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:06 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:58:06.423 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:06 smithi083 ceph-mon[24738]: purged_snaps scrub starts 2024-03-04T15:58:06.423 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:06 smithi083 ceph-mon[24738]: purged_snaps scrub ok 2024-03-04T15:58:06.423 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:06 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:58:06.423 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:06 smithi083 ceph-mon[24738]: osd.0 [v2:172.21.15.83:6802/3013656475,v1:172.21.15.83:6803/3013656475] boot 2024-03-04T15:58:06.423 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:06 smithi083 ceph-mon[24738]: osdmap e8: 1 total, 1 up, 1 in 2024-03-04T15:58:06.423 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:06 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:58:06.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:06 smithi086 ceph-mon[25584]: purged_snaps scrub starts 2024-03-04T15:58:06.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:06 smithi086 ceph-mon[25584]: purged_snaps scrub ok 2024-03-04T15:58:06.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:06 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:58:06.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:06 smithi086 ceph-mon[25584]: osd.0 [v2:172.21.15.83:6802/3013656475,v1:172.21.15.83:6803/3013656475] boot 2024-03-04T15:58:06.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:06 smithi086 ceph-mon[25584]: osdmap e8: 1 total, 1 up, 1 in 2024-03-04T15:58:06.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:06 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:58:07.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:07 smithi086 ceph-mon[25584]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T15:58:07.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:07 smithi086 ceph-mon[25584]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:07.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:07 smithi083 ceph-mon[29452]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T15:58:07.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:07 smithi083 ceph-mon[29452]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:07.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:07 smithi083 ceph-mon[24738]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T15:58:07.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:07 smithi083 ceph-mon[24738]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:09.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:08 smithi086 ceph-mon[25584]: Detected new or changed devices on smithi083 2024-03-04T15:58:09.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:08 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:09.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:08 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:09.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:08 smithi086 ceph-mon[25584]: Adjusting osd_memory_target on smithi083 to 16064M 2024-03-04T15:58:09.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:08 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:09.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:08 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:09.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:08 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:09.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:08 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:09.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:08 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:09.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:08 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:09.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[24738]: Detected new or changed devices on smithi083 2024-03-04T15:58:09.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:09.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:09.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[24738]: Adjusting osd_memory_target on smithi083 to 16064M 2024-03-04T15:58:09.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:09.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:09.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:09.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:09.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:09.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:09.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[29452]: Detected new or changed devices on smithi083 2024-03-04T15:58:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[29452]: Adjusting osd_memory_target on smithi083 to 16064M 2024-03-04T15:58:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:08 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:10.077 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:09 smithi083 ceph-mon[24738]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:10.077 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:09 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:58:10.077 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:09 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:58:10.078 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:09 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:10.078 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:09 smithi083 ceph-mon[29452]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:10.078 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:09 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:58:10.078 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:09 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:58:10.078 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:09 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:10.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:09 smithi086 ceph-mon[25584]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:10.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:09 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:58:10.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:09 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:58:10.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:09 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:11.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:10 smithi086 ceph-mon[25584]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:11.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:10 smithi083 ceph-mon[24738]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:11.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:10 smithi083 ceph-mon[29452]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:12.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:11 smithi086 ceph-mon[25584]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:12.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:11 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3851540979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c23dd2-2738-4b9b-8155-720d51c5a16c"}]: dispatch 2024-03-04T15:58:12.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:11 smithi086 ceph-mon[25584]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c23dd2-2738-4b9b-8155-720d51c5a16c"}]: dispatch 2024-03-04T15:58:12.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:11 smithi086 ceph-mon[25584]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50c23dd2-2738-4b9b-8155-720d51c5a16c"}]': finished 2024-03-04T15:58:12.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:11 smithi086 ceph-mon[25584]: osdmap e10: 2 total, 1 up, 2 in 2024-03-04T15:58:12.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:11 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:58:12.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:11 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/2263336472' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:58:12.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:11 smithi083 ceph-mon[29452]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:12.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:11 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3851540979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c23dd2-2738-4b9b-8155-720d51c5a16c"}]: dispatch 2024-03-04T15:58:12.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:11 smithi083 ceph-mon[29452]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c23dd2-2738-4b9b-8155-720d51c5a16c"}]: dispatch 2024-03-04T15:58:12.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:11 smithi083 ceph-mon[29452]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50c23dd2-2738-4b9b-8155-720d51c5a16c"}]': finished 2024-03-04T15:58:12.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:11 smithi083 ceph-mon[29452]: osdmap e10: 2 total, 1 up, 2 in 2024-03-04T15:58:12.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:11 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:58:12.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:11 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/2263336472' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:58:12.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:11 smithi083 ceph-mon[24738]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:12.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:11 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3851540979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c23dd2-2738-4b9b-8155-720d51c5a16c"}]: dispatch 2024-03-04T15:58:12.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:11 smithi083 ceph-mon[24738]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c23dd2-2738-4b9b-8155-720d51c5a16c"}]: dispatch 2024-03-04T15:58:12.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:11 smithi083 ceph-mon[24738]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50c23dd2-2738-4b9b-8155-720d51c5a16c"}]': finished 2024-03-04T15:58:12.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:11 smithi083 ceph-mon[24738]: osdmap e10: 2 total, 1 up, 2 in 2024-03-04T15:58:12.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:11 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:58:12.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:11 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2263336472' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:58:14.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:13 smithi086 ceph-mon[25584]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:14.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:13 smithi083 ceph-mon[29452]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:14.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:13 smithi083 ceph-mon[24738]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:15.868 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:15 smithi083 ceph-mon[24738]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:15.868 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:15 smithi083 ceph-mon[29452]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:16.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:15 smithi086 ceph-mon[25584]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:17.928 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:17 smithi083 ceph-mon[24738]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:17.928 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:17 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T15:58:17.928 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:17 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:17.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:17 smithi083 ceph-mon[29452]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:17.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:17 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T15:58:17.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:17 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:18.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:17 smithi086 ceph-mon[25584]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:18.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:17 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T15:58:18.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:17 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:18.924 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:18 smithi083 ceph-mon[24738]: Deploying daemon osd.1 on smithi083 2024-03-04T15:58:18.925 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:18 smithi083 ceph-mon[29452]: Deploying daemon osd.1 on smithi083 2024-03-04T15:58:19.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:18 smithi086 ceph-mon[25584]: Deploying daemon osd.1 on smithi083 2024-03-04T15:58:19.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:19 smithi083 ceph-mon[24738]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:19.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:19 smithi083 ceph-mon[29452]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:20.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:19 smithi086 ceph-mon[25584]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:21.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:21 smithi083 ceph-mon[24738]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:21.219 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:21 smithi083 ceph-mon[29452]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:21.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:21 smithi086 ceph-mon[25584]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:23.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:23 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:23.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:23 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:23.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:23 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:23.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:23 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:23.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:23 smithi083 ceph-mon[29452]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:23.482 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:23 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:23.482 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:23 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:23.482 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:23 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:23.482 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:23 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:23.482 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:23 smithi083 ceph-mon[24738]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:23.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:23 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:23.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:23 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:23.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:23 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:23.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:23 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:23.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:23 smithi086 ceph-mon[25584]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:24.894 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 1 on host 'smithi083' 2024-03-04T15:58:25.474 DEBUG:teuthology.orchestra.run.smithi083:osd.1> sudo journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.1.service 2024-03-04T15:58:25.476 INFO:tasks.cephadm:Deploying osd.2 on smithi083 with /dev/vg_nvme/lv_2... 2024-03-04T15:58:25.476 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-04T15:58:26.074 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:25 smithi083 ceph-mon[24738]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:26.075 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:25 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:26.075 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:25 smithi083 ceph-mon[24738]: from='osd.1 [v2:172.21.15.83:6810/2964268155,v1:172.21.15.83:6811/2964268155]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:58:26.075 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:25 smithi083 ceph-mon[24738]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:58:26.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:25 smithi083 ceph-mon[29452]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:26.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:25 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:26.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:25 smithi083 ceph-mon[29452]: from='osd.1 [v2:172.21.15.83:6810/2964268155,v1:172.21.15.83:6811/2964268155]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:58:26.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:25 smithi083 ceph-mon[29452]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:58:26.188 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:25 smithi086 ceph-mon[25584]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:26.188 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:25 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:26.188 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:25 smithi086 ceph-mon[25584]: from='osd.1 [v2:172.21.15.83:6810/2964268155,v1:172.21.15.83:6811/2964268155]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:58:26.188 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:25 smithi086 ceph-mon[25584]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:58:27.119 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:27 smithi083 ceph-mon[29452]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T15:58:27.120 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:27 smithi083 ceph-mon[29452]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T15:58:27.120 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:27 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:58:27.120 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:27 smithi083 ceph-mon[29452]: from='osd.1 [v2:172.21.15.83:6810/2964268155,v1:172.21.15.83:6811/2964268155]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T15:58:27.120 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:27 smithi083 ceph-mon[29452]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T15:58:27.120 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:27 smithi083 ceph-mon[29452]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:27.120 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:27 smithi083 ceph-mon[24738]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T15:58:27.121 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:27 smithi083 ceph-mon[24738]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T15:58:27.121 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:27 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:58:27.121 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:27 smithi083 ceph-mon[24738]: from='osd.1 [v2:172.21.15.83:6810/2964268155,v1:172.21.15.83:6811/2964268155]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T15:58:27.121 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:27 smithi083 ceph-mon[24738]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T15:58:27.121 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:27 smithi083 ceph-mon[24738]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:27.121 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 15:58:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1[35256]: 2024-03-04T15:58:27.058+0000 7fb319001700 -1 osd.1 0 waiting for initial osdmap 2024-03-04T15:58:27.121 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 15:58:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1[35256]: 2024-03-04T15:58:27.060+0000 7fb311994700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T15:58:27.326 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:27 smithi086 ceph-mon[25584]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T15:58:27.327 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:27 smithi086 ceph-mon[25584]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T15:58:27.327 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:27 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:58:27.327 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:27 smithi086 ceph-mon[25584]: from='osd.1 [v2:172.21.15.83:6810/2964268155,v1:172.21.15.83:6811/2964268155]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T15:58:27.327 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:27 smithi086 ceph-mon[25584]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T15:58:27.327 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:27 smithi086 ceph-mon[25584]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:27.610 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-04T15:58:27.610 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-04T15:58:27.610 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-03-04T15:58:27.610 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-03-04T15:58:27.610 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0151392 s, 693 MB/s 2024-03-04T15:58:27.611 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-03-04T15:58:28.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:28 smithi083 ceph-mon[24738]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-04T15:58:28.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:28 smithi083 ceph-mon[24738]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T15:58:28.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:28 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:58:28.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:28 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:58:28.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:28 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:28.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:28 smithi083 ceph-mon[29452]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-04T15:58:28.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:28 smithi083 ceph-mon[29452]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T15:58:28.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:28 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:58:28.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:28 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:58:28.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:28 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:28.306 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi083:vg_nvme/lv_2 2024-03-04T15:58:28.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:28 smithi086 ceph-mon[25584]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-04T15:58:28.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:28 smithi086 ceph-mon[25584]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T15:58:28.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:28 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:58:28.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:28 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:58:28.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:28 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:29.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:29 smithi083 ceph-mon[29452]: purged_snaps scrub starts 2024-03-04T15:58:29.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:29 smithi083 ceph-mon[29452]: purged_snaps scrub ok 2024-03-04T15:58:29.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:29 smithi083 ceph-mon[29452]: osd.1 [v2:172.21.15.83:6810/2964268155,v1:172.21.15.83:6811/2964268155] boot 2024-03-04T15:58:29.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:29 smithi083 ceph-mon[29452]: osdmap e13: 2 total, 2 up, 2 in 2024-03-04T15:58:29.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:29 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:58:29.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:29 smithi083 ceph-mon[29452]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:29.348 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:29 smithi083 ceph-mon[24738]: purged_snaps scrub starts 2024-03-04T15:58:29.348 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:29 smithi083 ceph-mon[24738]: purged_snaps scrub ok 2024-03-04T15:58:29.348 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:29 smithi083 ceph-mon[24738]: osd.1 [v2:172.21.15.83:6810/2964268155,v1:172.21.15.83:6811/2964268155] boot 2024-03-04T15:58:29.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:29 smithi083 ceph-mon[24738]: osdmap e13: 2 total, 2 up, 2 in 2024-03-04T15:58:29.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:29 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:58:29.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:29 smithi083 ceph-mon[24738]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:29.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:29 smithi086 ceph-mon[25584]: purged_snaps scrub starts 2024-03-04T15:58:29.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:29 smithi086 ceph-mon[25584]: purged_snaps scrub ok 2024-03-04T15:58:29.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:29 smithi086 ceph-mon[25584]: osd.1 [v2:172.21.15.83:6810/2964268155,v1:172.21.15.83:6811/2964268155] boot 2024-03-04T15:58:29.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:29 smithi086 ceph-mon[25584]: osdmap e13: 2 total, 2 up, 2 in 2024-03-04T15:58:29.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:29 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:58:29.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:29 smithi086 ceph-mon[25584]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:58:30.280 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:30 smithi083 ceph-mon[24738]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T15:58:30.280 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:30 smithi083 ceph-mon[29452]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T15:58:30.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:30 smithi086 ceph-mon[25584]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T15:58:31.227 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:31 smithi083 ceph-mon[24738]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:31.228 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:31 smithi083 ceph-mon[29452]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:31.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:31 smithi086 ceph-mon[25584]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:32.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:32 smithi083 ceph-mon[24738]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:32.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:32 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:58:32.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:32 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:58:32.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:32 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:32.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:32 smithi083 ceph-mon[29452]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:32.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:32 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:58:32.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:32 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:58:32.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:32 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:32.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:32 smithi086 ceph-mon[25584]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:32.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:32 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:58:32.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:32 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:58:32.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:32 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:33.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[29452]: Detected new or changed devices on smithi083 2024-03-04T15:58:33.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:33.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:33.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:33.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[29452]: Adjusting osd_memory_target on smithi083 to 8032M 2024-03-04T15:58:33.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:33.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:33.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:33.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:33.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:33.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:33.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[29452]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:33.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[24738]: Detected new or changed devices on smithi083 2024-03-04T15:58:33.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:33.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:33.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:33.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[24738]: Adjusting osd_memory_target on smithi083 to 8032M 2024-03-04T15:58:33.493 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:33.493 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:33.493 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:33.493 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:33.493 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:33.493 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:33.494 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:33 smithi083 ceph-mon[24738]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:33.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:33 smithi086 ceph-mon[25584]: Detected new or changed devices on smithi083 2024-03-04T15:58:33.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:33 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:33.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:33 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:33.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:33 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:33.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:33 smithi086 ceph-mon[25584]: Adjusting osd_memory_target on smithi083 to 8032M 2024-03-04T15:58:33.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:33 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:33.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:33 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:33.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:33 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:33.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:33 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:33.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:33 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:33.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:33 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:33.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:33 smithi086 ceph-mon[25584]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:34.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:34 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3348590619' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6bf2ebd9-d898-49c4-befd-df999267dab5"}]: dispatch 2024-03-04T15:58:34.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:34 smithi086 ceph-mon[25584]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6bf2ebd9-d898-49c4-befd-df999267dab5"}]: dispatch 2024-03-04T15:58:34.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:34 smithi086 ceph-mon[25584]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6bf2ebd9-d898-49c4-befd-df999267dab5"}]': finished 2024-03-04T15:58:34.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:34 smithi086 ceph-mon[25584]: osdmap e15: 3 total, 2 up, 3 in 2024-03-04T15:58:34.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:34 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:34.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:34 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3348590619' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6bf2ebd9-d898-49c4-befd-df999267dab5"}]: dispatch 2024-03-04T15:58:34.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:34 smithi083 ceph-mon[24738]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6bf2ebd9-d898-49c4-befd-df999267dab5"}]: dispatch 2024-03-04T15:58:34.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:34 smithi083 ceph-mon[24738]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6bf2ebd9-d898-49c4-befd-df999267dab5"}]': finished 2024-03-04T15:58:34.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:34 smithi083 ceph-mon[24738]: osdmap e15: 3 total, 2 up, 3 in 2024-03-04T15:58:34.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:34 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:34.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:34 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3348590619' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6bf2ebd9-d898-49c4-befd-df999267dab5"}]: dispatch 2024-03-04T15:58:34.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:34 smithi083 ceph-mon[29452]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6bf2ebd9-d898-49c4-befd-df999267dab5"}]: dispatch 2024-03-04T15:58:34.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:34 smithi083 ceph-mon[29452]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6bf2ebd9-d898-49c4-befd-df999267dab5"}]': finished 2024-03-04T15:58:34.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:34 smithi083 ceph-mon[29452]: osdmap e15: 3 total, 2 up, 3 in 2024-03-04T15:58:34.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:34 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:35.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:35 smithi086 ceph-mon[25584]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:35.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:35 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1142688211' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:58:35.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:35 smithi083 ceph-mon[24738]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:35.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:35 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1142688211' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:58:35.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:35 smithi083 ceph-mon[29452]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:35.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:35 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1142688211' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:58:37.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:37 smithi083 ceph-mon[24738]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:37.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:37 smithi083 ceph-mon[29452]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:38.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:37 smithi086 ceph-mon[25584]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:39.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:39 smithi083 ceph-mon[24738]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:39.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:39 smithi083 ceph-mon[29452]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:40.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:39 smithi086 ceph-mon[25584]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:41.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:41 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T15:58:41.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:41 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:41.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:41 smithi083 ceph-mon[24738]: Deploying daemon osd.2 on smithi083 2024-03-04T15:58:41.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:41 smithi083 ceph-mon[24738]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:41.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:41 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T15:58:41.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:41 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:41.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:41 smithi083 ceph-mon[29452]: Deploying daemon osd.2 on smithi083 2024-03-04T15:58:41.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:41 smithi083 ceph-mon[29452]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:41.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:41 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T15:58:41.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:41 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:41.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:41 smithi086 ceph-mon[25584]: Deploying daemon osd.2 on smithi083 2024-03-04T15:58:41.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:41 smithi086 ceph-mon[25584]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:44.117 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:43 smithi083 ceph-mon[24738]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:44.118 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:43 smithi083 ceph-mon[29452]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:44.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:43 smithi086 ceph-mon[25584]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:45.074 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:44 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:58:45.074 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:44 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:58:45.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:44 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:58:45.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:44 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:58:45.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:44 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:58:45.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:44 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:58:45.899 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:45 smithi083 ceph-mon[24738]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:45.900 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:45.900 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:45.900 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:45.900 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:45.900 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:45 smithi083 ceph-mon[29452]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:45.900 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:45 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:45.900 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:45 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:45.900 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:45 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:45.901 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:45 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:46.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:45 smithi086 ceph-mon[25584]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:46.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:45 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:46.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:45 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:46.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:45 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:46.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:45 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:48.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:47 smithi086 ceph-mon[25584]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:48.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:47 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:48.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:47 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:48.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:47 smithi083 ceph-mon[24738]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:48.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:47 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:48.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:47 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:48.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:47 smithi083 ceph-mon[29452]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:48.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:47 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:48.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:47 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:48.469 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 2 on host 'smithi083' 2024-03-04T15:58:49.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:48 smithi083 ceph-mon[24738]: from='osd.2 [v2:172.21.15.83:6818/1243652505,v1:172.21.15.83:6819/1243652505]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T15:58:49.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:48 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:49.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:48 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:49.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:48 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:49.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:48 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:49.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:48 smithi083 ceph-mon[29452]: from='osd.2 [v2:172.21.15.83:6818/1243652505,v1:172.21.15.83:6819/1243652505]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T15:58:49.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:48 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:49.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:48 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:49.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:48 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:49.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:48 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:49.177 DEBUG:teuthology.orchestra.run.smithi083:osd.2> sudo journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.2.service 2024-03-04T15:58:49.179 INFO:tasks.cephadm:Deploying osd.3 on smithi083 with /dev/vg_nvme/lv_1... 2024-03-04T15:58:49.180 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-04T15:58:49.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:48 smithi086 ceph-mon[25584]: from='osd.2 [v2:172.21.15.83:6818/1243652505,v1:172.21.15.83:6819/1243652505]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T15:58:49.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:48 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:49.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:48 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:49.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:48 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:49.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:48 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:50.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:49 smithi086 ceph-mon[25584]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:50.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:49 smithi086 ceph-mon[25584]: from='osd.2 [v2:172.21.15.83:6818/1243652505,v1:172.21.15.83:6819/1243652505]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T15:58:50.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:49 smithi086 ceph-mon[25584]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T15:58:50.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:49 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:50.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:49 smithi086 ceph-mon[25584]: from='osd.2 [v2:172.21.15.83:6818/1243652505,v1:172.21.15.83:6819/1243652505]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T15:58:50.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:49 smithi083 ceph-mon[29452]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:50.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:49 smithi083 ceph-mon[29452]: from='osd.2 [v2:172.21.15.83:6818/1243652505,v1:172.21.15.83:6819/1243652505]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T15:58:50.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:49 smithi083 ceph-mon[29452]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T15:58:50.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:49 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:50.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:49 smithi083 ceph-mon[29452]: from='osd.2 [v2:172.21.15.83:6818/1243652505,v1:172.21.15.83:6819/1243652505]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T15:58:50.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:49 smithi083 ceph-mon[24738]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:50.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:49 smithi083 ceph-mon[24738]: from='osd.2 [v2:172.21.15.83:6818/1243652505,v1:172.21.15.83:6819/1243652505]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T15:58:50.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:49 smithi083 ceph-mon[24738]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T15:58:50.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:49 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:50.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:49 smithi083 ceph-mon[24738]: from='osd.2 [v2:172.21.15.83:6818/1243652505,v1:172.21.15.83:6819/1243652505]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T15:58:50.242 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 15:58:49 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2[38193]: 2024-03-04T15:58:49.836+0000 7f5277329700 -1 osd.2 0 waiting for initial osdmap 2024-03-04T15:58:50.242 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 15:58:49 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2[38193]: 2024-03-04T15:58:49.839+0000 7f52724c1700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T15:58:50.836 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:50 smithi083 ceph-mon[24738]: from='osd.2 [v2:172.21.15.83:6818/1243652505,v1:172.21.15.83:6819/1243652505]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-04T15:58:50.836 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:50 smithi083 ceph-mon[24738]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T15:58:51.181 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:50 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:51.181 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:50 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:51.181 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:50 smithi083 ceph-mon[29452]: from='osd.2 [v2:172.21.15.83:6818/1243652505,v1:172.21.15.83:6819/1243652505]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-04T15:58:51.181 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:50 smithi083 ceph-mon[29452]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T15:58:51.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:50 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:51.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:50 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:51.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:50 smithi086 ceph-mon[25584]: from='osd.2 [v2:172.21.15.83:6818/1243652505,v1:172.21.15.83:6819/1243652505]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-04T15:58:51.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:50 smithi086 ceph-mon[25584]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T15:58:51.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:50 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:51.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:50 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:51.860 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:51 smithi083 ceph-mon[29452]: purged_snaps scrub starts 2024-03-04T15:58:51.860 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:51 smithi083 ceph-mon[29452]: purged_snaps scrub ok 2024-03-04T15:58:51.860 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:51 smithi083 ceph-mon[29452]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:51.861 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:51 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:51.861 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:51 smithi083 ceph-mon[29452]: osd.2 [v2:172.21.15.83:6818/1243652505,v1:172.21.15.83:6819/1243652505] boot 2024-03-04T15:58:51.861 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:51 smithi083 ceph-mon[29452]: osdmap e18: 3 total, 3 up, 3 in 2024-03-04T15:58:51.861 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:51 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:51.861 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:51 smithi083 ceph-mon[24738]: purged_snaps scrub starts 2024-03-04T15:58:51.861 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:51 smithi083 ceph-mon[24738]: purged_snaps scrub ok 2024-03-04T15:58:51.861 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:51 smithi083 ceph-mon[24738]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:51.861 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:51 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:51.861 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:51 smithi083 ceph-mon[24738]: osd.2 [v2:172.21.15.83:6818/1243652505,v1:172.21.15.83:6819/1243652505] boot 2024-03-04T15:58:51.861 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:51 smithi083 ceph-mon[24738]: osdmap e18: 3 total, 3 up, 3 in 2024-03-04T15:58:51.861 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:51 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:52.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:51 smithi086 ceph-mon[25584]: purged_snaps scrub starts 2024-03-04T15:58:52.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:51 smithi086 ceph-mon[25584]: purged_snaps scrub ok 2024-03-04T15:58:52.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:51 smithi086 ceph-mon[25584]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:58:52.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:51 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:52.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:51 smithi086 ceph-mon[25584]: osd.2 [v2:172.21.15.83:6818/1243652505,v1:172.21.15.83:6819/1243652505] boot 2024-03-04T15:58:52.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:51 smithi086 ceph-mon[25584]: osdmap e18: 3 total, 3 up, 3 in 2024-03-04T15:58:52.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:51 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:58:52.651 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-04T15:58:52.652 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-04T15:58:52.652 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-03-04T15:58:52.652 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-03-04T15:58:52.652 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0133465 s, 786 MB/s 2024-03-04T15:58:52.652 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-03-04T15:58:53.162 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:53 smithi083 ceph-mon[29452]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T15:58:53.162 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:53 smithi083 ceph-mon[29452]: pgmap v73: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:53.162 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:53 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-04T15:58:53.162 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:53 smithi083 ceph-mon[29452]: Detected new or changed devices on smithi083 2024-03-04T15:58:53.163 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:53 smithi083 ceph-mon[24738]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T15:58:53.163 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:53 smithi083 ceph-mon[24738]: pgmap v73: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:53.163 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:53 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-04T15:58:53.163 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:53 smithi083 ceph-mon[24738]: Detected new or changed devices on smithi083 2024-03-04T15:58:53.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:53 smithi086 ceph-mon[25584]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T15:58:53.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:53 smithi086 ceph-mon[25584]: pgmap v73: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:53.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:53 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-04T15:58:53.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:53 smithi086 ceph-mon[25584]: Detected new or changed devices on smithi083 2024-03-04T15:58:53.650 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi083:vg_nvme/lv_1 2024-03-04T15:58:54.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:54.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:54.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:54.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:54.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[24738]: Adjusting osd_memory_target on smithi083 to 5354M 2024-03-04T15:58:54.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-04T15:58:54.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[24738]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T15:58:54.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:54.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T15:58:54.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:54.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:54.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:54.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:54.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:54.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[29452]: Adjusting osd_memory_target on smithi083 to 5354M 2024-03-04T15:58:54.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-04T15:58:54.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[29452]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T15:58:54.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:54.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T15:58:54.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:54 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:54.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:54 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:54.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:54 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:54.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:54 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:54.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:54 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:54.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:54 smithi086 ceph-mon[25584]: Adjusting osd_memory_target on smithi083 to 5354M 2024-03-04T15:58:54.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:54 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-04T15:58:54.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:54 smithi086 ceph-mon[25584]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T15:58:54.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:54 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:54.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:54 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T15:58:54.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:54 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:54.888 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 15:58:54 smithi083 sudo[39487]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T15:58:54.888 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 15:58:54 smithi083 sudo[39487]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-04T15:58:54.888 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 15:58:54 smithi083 sudo[39487]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:55.160 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:55 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T15:58:55.160 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:55 smithi083 ceph-mon[29452]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T15:58:55.160 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:55 smithi083 ceph-mon[29452]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:55.160 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:55 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:55.161 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:55 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T15:58:55.161 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:55 smithi083 ceph-mon[24738]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T15:58:55.161 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:55 smithi083 ceph-mon[24738]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:55.161 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:55 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:55.161 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 15:58:54 smithi083 sudo[39490]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T15:58:55.161 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 15:58:54 smithi083 sudo[39490]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-04T15:58:55.161 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 15:58:54 smithi083 sudo[39490]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:55.427 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 15:58:55 smithi083 sudo[39511]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T15:58:55.427 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 15:58:55 smithi083 sudo[39511]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-04T15:58:55.427 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 15:58:55 smithi083 sudo[39511]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:55.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:55 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T15:58:55.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:55 smithi086 ceph-mon[25584]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T15:58:55.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:55 smithi086 ceph-mon[25584]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:55.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:55 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:55.683 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 15:58:55 smithi083 sudo[39540]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T15:58:55.683 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 15:58:55 smithi083 sudo[39540]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-04T15:58:55.683 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 15:58:55 smithi083 sudo[39540]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:55.948 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 15:58:55 smithi083 sudo[39580]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T15:58:55.948 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 15:58:55 smithi083 sudo[39580]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-04T15:58:55.948 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 15:58:55 smithi083 sudo[39580]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:56.210 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:56 smithi083 sudo[39632]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T15:58:56.210 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:56 smithi083 sudo[39632]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-04T15:58:56.210 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:56 smithi083 sudo[39632]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:56.211 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 15:58:55 smithi083 sudo[39623]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T15:58:56.211 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 15:58:55 smithi083 sudo[39623]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-04T15:58:56.211 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 15:58:55 smithi083 sudo[39623]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:56.460 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:56 smithi083 sudo[39646]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T15:58:56.460 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:56 smithi083 sudo[39646]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-04T15:58:56.460 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:56 smithi083 sudo[39646]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:56.714 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:56 smithi083 ceph-mon[29452]: osdmap e22: 3 total, 3 up, 3 in 2024-03-04T15:58:56.714 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:56 smithi083 ceph-mon[29452]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:56.714 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:56 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:58:56.714 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:56 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:58:56.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:56 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:56.715 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:56 smithi083 ceph-mon[24738]: osdmap e22: 3 total, 3 up, 3 in 2024-03-04T15:58:56.715 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:56 smithi083 ceph-mon[24738]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:56.715 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:56 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:58:56.715 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:56 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:58:56.716 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:56 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:56.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:56 smithi086 ceph-mon[25584]: osdmap e22: 3 total, 3 up, 3 in 2024-03-04T15:58:56.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:56 smithi086 ceph-mon[25584]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:56.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:56 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:58:56.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:56 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:58:56.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:56 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:56.972 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:56 smithi083 sudo[39652]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T15:58:56.972 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:56 smithi083 sudo[39652]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-04T15:58:56.972 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:56 smithi083 sudo[39652]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:57.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:56 smithi083 sudo[39659]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T15:58:57.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:56 smithi083 sudo[39659]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-04T15:58:57.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:56 smithi083 sudo[39659]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:57.470 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 sudo[27018]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T15:58:57.470 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 sudo[27018]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-04T15:58:57.470 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 sudo[27018]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:57.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 sudo[27021]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T15:58:57.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 sudo[27021]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-04T15:58:57.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 sudo[27021]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:58:57.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 ceph-mon[25584]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:58:57.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 ceph-mon[25584]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:58:57.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:58:57.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:57.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:58:57.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 ceph-mon[25584]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:58:57.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 ceph-mon[25584]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-04T15:58:57.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:58:57.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:57.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:58:57.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 ceph-mon[25584]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:57.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 ceph-mon[25584]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:58:57.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:58:57.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:57.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:57 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:58:57.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[24738]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:58:57.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[24738]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:58:57.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:58:57.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:57.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:58:57.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[24738]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:58:57.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[24738]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-04T15:58:57.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:58:57.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:57.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:58:57.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[24738]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:57.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[24738]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:58:57.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:58:57.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:57.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:58:57.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[29452]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:58:57.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[29452]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:58:57.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:58:57.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:57.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:58:57.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[29452]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:58:57.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[29452]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-04T15:58:57.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:58:57.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:57.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:58:57.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[29452]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:57.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[29452]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:58:57.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:58:57.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:57.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:57 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:58:58.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:58 smithi086 ceph-mon[25584]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:58:58.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:58 smithi086 ceph-mon[25584]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:58:58.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:58 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:58.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:58 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1932199689' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24508e11-465f-43e0-8aad-4ce41214b4b4"}]: dispatch 2024-03-04T15:58:58.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:58 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1932199689' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24508e11-465f-43e0-8aad-4ce41214b4b4"}]': finished 2024-03-04T15:58:58.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:58 smithi086 ceph-mon[25584]: osdmap e23: 4 total, 3 up, 4 in 2024-03-04T15:58:58.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:58 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:58:58.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:58 smithi083 ceph-mon[24738]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:58:58.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:58 smithi083 ceph-mon[24738]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:58:58.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:58 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:58.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:58 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1932199689' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24508e11-465f-43e0-8aad-4ce41214b4b4"}]: dispatch 2024-03-04T15:58:58.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:58 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1932199689' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24508e11-465f-43e0-8aad-4ce41214b4b4"}]': finished 2024-03-04T15:58:58.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:58 smithi083 ceph-mon[24738]: osdmap e23: 4 total, 3 up, 4 in 2024-03-04T15:58:58.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:58 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:58:58.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:58 smithi083 ceph-mon[29452]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:58:58.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:58 smithi083 ceph-mon[29452]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:58:58.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:58 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:58:58.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:58 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1932199689' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24508e11-465f-43e0-8aad-4ce41214b4b4"}]: dispatch 2024-03-04T15:58:58.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:58 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1932199689' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24508e11-465f-43e0-8aad-4ce41214b4b4"}]': finished 2024-03-04T15:58:58.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:58 smithi083 ceph-mon[29452]: osdmap e23: 4 total, 3 up, 4 in 2024-03-04T15:58:58.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:58 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:58:59.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:59 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1026109849' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:58:59.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:58:59 smithi086 ceph-mon[25584]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:59.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:59 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1026109849' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:58:59.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:58:59 smithi083 ceph-mon[24738]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:58:59.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:59 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1026109849' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:58:59.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:58:59 smithi083 ceph-mon[29452]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:01.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:01 smithi083 ceph-mon[24738]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:01.349 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:01 smithi083 ceph-mon[29452]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:01.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:01 smithi086 ceph-mon[25584]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:04.091 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:03 smithi083 ceph-mon[29452]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:04.091 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:03 smithi083 ceph-mon[24738]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:04.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:03 smithi086 ceph-mon[25584]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:05.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:04 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T15:59:05.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:04 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:05.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:04 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T15:59:05.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:04 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:05.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:04 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T15:59:05.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:04 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:05.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:05 smithi083 ceph-mon[24738]: Deploying daemon osd.3 on smithi083 2024-03-04T15:59:05.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:05 smithi083 ceph-mon[24738]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:05.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:05 smithi083 ceph-mon[29452]: Deploying daemon osd.3 on smithi083 2024-03-04T15:59:05.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:05 smithi083 ceph-mon[29452]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:06.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:05 smithi086 ceph-mon[25584]: Deploying daemon osd.3 on smithi083 2024-03-04T15:59:06.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:05 smithi086 ceph-mon[25584]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:07 smithi086 ceph-mon[25584]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:08.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:07 smithi083 ceph-mon[24738]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:08.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:07 smithi083 ceph-mon[29452]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:09.949 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:09 smithi083 ceph-mon[29452]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:09.950 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:09 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:09.950 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:09 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:09.950 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:09 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:09.950 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:09 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:09.950 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:09 smithi083 ceph-mon[24738]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:09.950 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:09 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:09.950 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:09 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:09.950 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:09 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:09.950 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:09 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:09.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:09 smithi086 ceph-mon[25584]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:09.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:09 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:09.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:09 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:09.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:09 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:09.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:09 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:11.203 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:11 smithi083 ceph-mon[24738]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:11.204 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:11 smithi083 ceph-mon[29452]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:11.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:11 smithi086 ceph-mon[25584]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:12.897 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 3 on host 'smithi083' 2024-03-04T15:59:13.185 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:13 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:13.185 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:13 smithi083 ceph-mon[24738]: from='osd.3 [v2:172.21.15.83:6826/1643785747,v1:172.21.15.83:6827/1643785747]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T15:59:13.185 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:13 smithi083 ceph-mon[24738]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T15:59:13.185 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:13 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:13.185 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:13 smithi083 ceph-mon[24738]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:13.185 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:13 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:13.185 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:13 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:13.185 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:13 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:13.185 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:13 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:13.186 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:13 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:13.186 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:13 smithi083 ceph-mon[29452]: from='osd.3 [v2:172.21.15.83:6826/1643785747,v1:172.21.15.83:6827/1643785747]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T15:59:13.186 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:13 smithi083 ceph-mon[29452]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T15:59:13.186 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:13 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:13.186 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:13 smithi083 ceph-mon[29452]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:13.186 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:13 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:13.186 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:13 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:13.186 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:13 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:13.186 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:13 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:13.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:13 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:13.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:13 smithi086 ceph-mon[25584]: from='osd.3 [v2:172.21.15.83:6826/1643785747,v1:172.21.15.83:6827/1643785747]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T15:59:13.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:13 smithi086 ceph-mon[25584]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T15:59:13.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:13 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:13.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:13 smithi086 ceph-mon[25584]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:13.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:13 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:13.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:13 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:13.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:13 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:13.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:13 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:13.537 DEBUG:teuthology.orchestra.run.smithi083:osd.3> sudo journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.3.service 2024-03-04T15:59:13.539 INFO:tasks.cephadm:Deploying osd.4 on smithi086 with /dev/vg_nvme/lv_4... 2024-03-04T15:59:13.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-04T15:59:14.408 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:14 smithi083 ceph-mon[24738]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-04T15:59:14.408 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:14 smithi083 ceph-mon[24738]: osdmap e24: 4 total, 3 up, 4 in 2024-03-04T15:59:14.408 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:14 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:59:14.408 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:14 smithi083 ceph-mon[24738]: from='osd.3 [v2:172.21.15.83:6826/1643785747,v1:172.21.15.83:6827/1643785747]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T15:59:14.408 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:14 smithi083 ceph-mon[24738]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T15:59:14.409 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:14 smithi083 ceph-mon[29452]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-04T15:59:14.409 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:14 smithi083 ceph-mon[29452]: osdmap e24: 4 total, 3 up, 4 in 2024-03-04T15:59:14.409 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:14 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:59:14.409 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:14 smithi083 ceph-mon[29452]: from='osd.3 [v2:172.21.15.83:6826/1643785747,v1:172.21.15.83:6827/1643785747]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T15:59:14.409 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:14 smithi083 ceph-mon[29452]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T15:59:14.409 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 15:59:14 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3[41162]: 2024-03-04T15:59:14.104+0000 7f2bc63b7700 -1 osd.3 0 waiting for initial osdmap 2024-03-04T15:59:14.409 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 15:59:14 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3[41162]: 2024-03-04T15:59:14.107+0000 7f2bc054d700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T15:59:14.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:14 smithi086 ceph-mon[25584]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-04T15:59:14.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:14 smithi086 ceph-mon[25584]: osdmap e24: 4 total, 3 up, 4 in 2024-03-04T15:59:14.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:14 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:59:14.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:14 smithi086 ceph-mon[25584]: from='osd.3 [v2:172.21.15.83:6826/1643785747,v1:172.21.15.83:6827/1643785747]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T15:59:14.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:14 smithi086 ceph-mon[25584]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T15:59:15.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:15 smithi086 ceph-mon[25584]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-04T15:59:15.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:15 smithi086 ceph-mon[25584]: osdmap e25: 4 total, 3 up, 4 in 2024-03-04T15:59:15.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:15 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:59:15.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:15 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:59:15.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:15 smithi086 ceph-mon[25584]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:15.400 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-04T15:59:15.400 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-04T15:59:15.400 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-03-04T15:59:15.400 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-03-04T15:59:15.400 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116589 s, 899 MB/s 2024-03-04T15:59:15.400 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-03-04T15:59:15.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:15 smithi083 ceph-mon[24738]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-04T15:59:15.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:15 smithi083 ceph-mon[24738]: osdmap e25: 4 total, 3 up, 4 in 2024-03-04T15:59:15.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:15 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:59:15.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:15 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:59:15.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:15 smithi083 ceph-mon[24738]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:15.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:15 smithi083 ceph-mon[29452]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-04T15:59:15.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:15 smithi083 ceph-mon[29452]: osdmap e25: 4 total, 3 up, 4 in 2024-03-04T15:59:15.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:15 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:59:15.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:15 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:59:15.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:15 smithi083 ceph-mon[29452]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:59:16.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi086:vg_nvme/lv_4 2024-03-04T15:59:16.399 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:16 smithi083 ceph-mon[29452]: purged_snaps scrub starts 2024-03-04T15:59:16.399 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:16 smithi083 ceph-mon[29452]: purged_snaps scrub ok 2024-03-04T15:59:16.399 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:16 smithi083 ceph-mon[29452]: osd.3 [v2:172.21.15.83:6826/1643785747,v1:172.21.15.83:6827/1643785747] boot 2024-03-04T15:59:16.399 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:16 smithi083 ceph-mon[29452]: osdmap e26: 4 total, 4 up, 4 in 2024-03-04T15:59:16.400 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:16 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:59:16.400 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:16 smithi083 ceph-mon[24738]: purged_snaps scrub starts 2024-03-04T15:59:16.400 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:16 smithi083 ceph-mon[24738]: purged_snaps scrub ok 2024-03-04T15:59:16.400 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:16 smithi083 ceph-mon[24738]: osd.3 [v2:172.21.15.83:6826/1643785747,v1:172.21.15.83:6827/1643785747] boot 2024-03-04T15:59:16.400 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:16 smithi083 ceph-mon[24738]: osdmap e26: 4 total, 4 up, 4 in 2024-03-04T15:59:16.400 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:16 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:59:16.420 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:16 smithi086 ceph-mon[25584]: purged_snaps scrub starts 2024-03-04T15:59:16.420 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:16 smithi086 ceph-mon[25584]: purged_snaps scrub ok 2024-03-04T15:59:16.421 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:16 smithi086 ceph-mon[25584]: osd.3 [v2:172.21.15.83:6826/1643785747,v1:172.21.15.83:6827/1643785747] boot 2024-03-04T15:59:16.421 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:16 smithi086 ceph-mon[25584]: osdmap e26: 4 total, 4 up, 4 in 2024-03-04T15:59:16.421 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:16 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T15:59:17.268 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:17 smithi086 ceph-mon[25584]: osdmap e27: 4 total, 4 up, 4 in 2024-03-04T15:59:17.269 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:17 smithi086 ceph-mon[25584]: Detected new or changed devices on smithi083 2024-03-04T15:59:17.269 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:17 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:17.269 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:17 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:17.269 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:17 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:17.269 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:17 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:17.269 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:17 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:17.269 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:17 smithi086 ceph-mon[25584]: Adjusting osd_memory_target on smithi083 to 4016M 2024-03-04T15:59:17.269 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:17 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:17.269 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:17 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:17.269 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:17 smithi086 ceph-mon[25584]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:17.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[24738]: osdmap e27: 4 total, 4 up, 4 in 2024-03-04T15:59:17.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[24738]: Detected new or changed devices on smithi083 2024-03-04T15:59:17.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:17.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:17.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:17.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:17.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:17.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[24738]: Adjusting osd_memory_target on smithi083 to 4016M 2024-03-04T15:59:17.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:17.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:17.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[24738]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:17.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[29452]: osdmap e27: 4 total, 4 up, 4 in 2024-03-04T15:59:17.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[29452]: Detected new or changed devices on smithi083 2024-03-04T15:59:17.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:17.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:17.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:17.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:17.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:17.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[29452]: Adjusting osd_memory_target on smithi083 to 4016M 2024-03-04T15:59:17.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:17.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:17.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:17 smithi083 ceph-mon[29452]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:18.632 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:18 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:59:18.632 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:18 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:59:18.633 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:18 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:18.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:18 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:59:18.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:18 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:59:18.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:18 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:18.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:18 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:59:18.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:18 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:59:18.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:18 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:19.488 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:19 smithi086 ceph-mon[25584]: from='client.34187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:59:19.489 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:19 smithi086 ceph-mon[25584]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:19.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:19 smithi083 ceph-mon[24738]: from='client.34187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:59:19.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:19 smithi083 ceph-mon[24738]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:19.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:19 smithi083 ceph-mon[29452]: from='client.34187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:59:19.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:19 smithi083 ceph-mon[29452]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:21.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:21 smithi086 ceph-mon[25584]: from='client.? 172.21.15.86:0/916078976' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79c9ab9a-0df7-485a-bba9-54470969695e"}]: dispatch 2024-03-04T15:59:21.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:21 smithi086 ceph-mon[25584]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79c9ab9a-0df7-485a-bba9-54470969695e"}]: dispatch 2024-03-04T15:59:21.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:21 smithi086 ceph-mon[25584]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79c9ab9a-0df7-485a-bba9-54470969695e"}]': finished 2024-03-04T15:59:21.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:21 smithi086 ceph-mon[25584]: osdmap e28: 5 total, 4 up, 5 in 2024-03-04T15:59:21.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:21 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:21.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:21 smithi086 ceph-mon[25584]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:21.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:21 smithi086 ceph-mon[25584]: from='client.? 172.21.15.86:0/963716779' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:59:21.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:21 smithi083 ceph-mon[24738]: from='client.? 172.21.15.86:0/916078976' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79c9ab9a-0df7-485a-bba9-54470969695e"}]: dispatch 2024-03-04T15:59:21.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:21 smithi083 ceph-mon[24738]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79c9ab9a-0df7-485a-bba9-54470969695e"}]: dispatch 2024-03-04T15:59:21.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:21 smithi083 ceph-mon[24738]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79c9ab9a-0df7-485a-bba9-54470969695e"}]': finished 2024-03-04T15:59:21.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:21 smithi083 ceph-mon[24738]: osdmap e28: 5 total, 4 up, 5 in 2024-03-04T15:59:21.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:21 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:21.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:21 smithi083 ceph-mon[24738]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:21.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:21 smithi083 ceph-mon[24738]: from='client.? 172.21.15.86:0/963716779' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:59:21.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:21 smithi083 ceph-mon[29452]: from='client.? 172.21.15.86:0/916078976' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79c9ab9a-0df7-485a-bba9-54470969695e"}]: dispatch 2024-03-04T15:59:21.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:21 smithi083 ceph-mon[29452]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79c9ab9a-0df7-485a-bba9-54470969695e"}]: dispatch 2024-03-04T15:59:21.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:21 smithi083 ceph-mon[29452]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79c9ab9a-0df7-485a-bba9-54470969695e"}]': finished 2024-03-04T15:59:21.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:21 smithi083 ceph-mon[29452]: osdmap e28: 5 total, 4 up, 5 in 2024-03-04T15:59:21.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:21 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:21.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:21 smithi083 ceph-mon[29452]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:21.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:21 smithi083 ceph-mon[29452]: from='client.? 172.21.15.86:0/963716779' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:59:23.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:23 smithi086 ceph-mon[25584]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:24.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:23 smithi083 ceph-mon[24738]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:24.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:23 smithi083 ceph-mon[29452]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:25.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:25 smithi086 ceph-mon[25584]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:26.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:25 smithi083 ceph-mon[24738]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:26.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:25 smithi083 ceph-mon[29452]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:27.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:26 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T15:59:27.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:26 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:27.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:26 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T15:59:27.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:26 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:27.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:26 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T15:59:27.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:26 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:27.922 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:27 smithi086 ceph-mon[25584]: Deploying daemon osd.4 on smithi086 2024-03-04T15:59:27.922 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:27 smithi086 ceph-mon[25584]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:28.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:27 smithi083 ceph-mon[24738]: Deploying daemon osd.4 on smithi086 2024-03-04T15:59:28.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:27 smithi083 ceph-mon[24738]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:28.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:27 smithi083 ceph-mon[29452]: Deploying daemon osd.4 on smithi086 2024-03-04T15:59:28.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:27 smithi083 ceph-mon[29452]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:30.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:29 smithi086 ceph-mon[25584]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:30.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:29 smithi083 ceph-mon[24738]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:30.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:29 smithi083 ceph-mon[29452]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:31.463 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:31 smithi086 ceph-mon[25584]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:31.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:31 smithi083 ceph-mon[24738]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:31.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:31 smithi083 ceph-mon[29452]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:32.353 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:32 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:32.353 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:32 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:32.353 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:32 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:32.353 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:32 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:32.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:32 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:32.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:32 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:32.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:32 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:32.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:32 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:32.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:32 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:32.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:32 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:32.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:32 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:32.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:32 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:33.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:33 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:33.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:33 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:33.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:33 smithi083 ceph-mon[29452]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:33.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:33 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:33.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:33 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:33.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:33 smithi083 ceph-mon[24738]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:34.158 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:33 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:34.158 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:33 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:34.158 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:33 smithi086 ceph-mon[25584]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:34.455 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 4 on host 'smithi086' 2024-03-04T15:59:35.036 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:34 smithi086 ceph-mon[25584]: from='osd.4 [v2:172.21.15.86:6800/745738669,v1:172.21.15.86:6801/745738669]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T15:59:35.036 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:34 smithi086 ceph-mon[25584]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T15:59:35.036 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:34 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:35.036 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:34 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:35.036 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:34 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:35.037 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:34 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:35.058 DEBUG:teuthology.orchestra.run.smithi086:osd.4> sudo journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.4.service 2024-03-04T15:59:35.061 INFO:tasks.cephadm:Deploying osd.5 on smithi086 with /dev/vg_nvme/lv_3... 2024-03-04T15:59:35.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-04T15:59:35.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:34 smithi083 ceph-mon[24738]: from='osd.4 [v2:172.21.15.86:6800/745738669,v1:172.21.15.86:6801/745738669]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T15:59:35.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:34 smithi083 ceph-mon[24738]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T15:59:35.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:34 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:35.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:34 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:35.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:34 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:35.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:34 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:35.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:34 smithi083 ceph-mon[29452]: from='osd.4 [v2:172.21.15.86:6800/745738669,v1:172.21.15.86:6801/745738669]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T15:59:35.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:34 smithi083 ceph-mon[29452]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T15:59:35.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:34 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:35.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:34 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:35.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:34 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:35.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:34 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:35.950 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:35 smithi086 ceph-mon[25584]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-04T15:59:35.950 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:35 smithi086 ceph-mon[25584]: osdmap e29: 5 total, 4 up, 5 in 2024-03-04T15:59:35.950 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:35 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:35.950 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:35 smithi086 ceph-mon[25584]: from='osd.4 [v2:172.21.15.86:6800/745738669,v1:172.21.15.86:6801/745738669]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T15:59:35.951 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:35 smithi086 ceph-mon[25584]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T15:59:35.951 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:35 smithi086 ceph-mon[25584]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:35.951 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 15:59:35 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[28935]: 2024-03-04T15:59:35.757+0000 7f900293d700 -1 osd.4 0 waiting for initial osdmap 2024-03-04T15:59:35.951 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 15:59:35 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[28935]: 2024-03-04T15:59:35.761+0000 7f8fff2d8700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T15:59:36.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:35 smithi083 ceph-mon[24738]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-04T15:59:36.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:35 smithi083 ceph-mon[24738]: osdmap e29: 5 total, 4 up, 5 in 2024-03-04T15:59:36.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:35 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:36.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:35 smithi083 ceph-mon[24738]: from='osd.4 [v2:172.21.15.86:6800/745738669,v1:172.21.15.86:6801/745738669]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T15:59:36.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:35 smithi083 ceph-mon[24738]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T15:59:36.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:35 smithi083 ceph-mon[24738]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:36.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:35 smithi083 ceph-mon[29452]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-04T15:59:36.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:35 smithi083 ceph-mon[29452]: osdmap e29: 5 total, 4 up, 5 in 2024-03-04T15:59:36.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:35 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:36.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:35 smithi083 ceph-mon[29452]: from='osd.4 [v2:172.21.15.86:6800/745738669,v1:172.21.15.86:6801/745738669]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T15:59:36.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:35 smithi083 ceph-mon[29452]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T15:59:36.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:35 smithi083 ceph-mon[29452]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-04T15:59:36.854 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:36 smithi086 ceph-mon[25584]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T15:59:36.854 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:36 smithi086 ceph-mon[25584]: osdmap e30: 5 total, 4 up, 5 in 2024-03-04T15:59:36.854 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:36 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:36.854 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:36 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:37.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:36 smithi083 ceph-mon[24738]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T15:59:37.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:36 smithi083 ceph-mon[24738]: osdmap e30: 5 total, 4 up, 5 in 2024-03-04T15:59:37.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:36 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:37.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:36 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:37.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:36 smithi083 ceph-mon[29452]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T15:59:37.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:36 smithi083 ceph-mon[29452]: osdmap e30: 5 total, 4 up, 5 in 2024-03-04T15:59:37.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:36 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:37.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:36 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:37.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:37 smithi086 ceph-mon[25584]: purged_snaps scrub starts 2024-03-04T15:59:37.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:37 smithi086 ceph-mon[25584]: purged_snaps scrub ok 2024-03-04T15:59:37.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:37 smithi086 ceph-mon[25584]: osd.4 [v2:172.21.15.86:6800/745738669,v1:172.21.15.86:6801/745738669] boot 2024-03-04T15:59:37.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:37 smithi086 ceph-mon[25584]: osdmap e31: 5 total, 5 up, 5 in 2024-03-04T15:59:37.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:37 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:37.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:37 smithi086 ceph-mon[25584]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:38.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:37 smithi083 ceph-mon[24738]: purged_snaps scrub starts 2024-03-04T15:59:38.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:37 smithi083 ceph-mon[24738]: purged_snaps scrub ok 2024-03-04T15:59:38.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:37 smithi083 ceph-mon[24738]: osd.4 [v2:172.21.15.86:6800/745738669,v1:172.21.15.86:6801/745738669] boot 2024-03-04T15:59:38.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:37 smithi083 ceph-mon[24738]: osdmap e31: 5 total, 5 up, 5 in 2024-03-04T15:59:38.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:37 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:38.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:37 smithi083 ceph-mon[24738]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:38.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:37 smithi083 ceph-mon[29452]: purged_snaps scrub starts 2024-03-04T15:59:38.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:37 smithi083 ceph-mon[29452]: purged_snaps scrub ok 2024-03-04T15:59:38.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:37 smithi083 ceph-mon[29452]: osd.4 [v2:172.21.15.86:6800/745738669,v1:172.21.15.86:6801/745738669] boot 2024-03-04T15:59:38.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:37 smithi083 ceph-mon[29452]: osdmap e31: 5 total, 5 up, 5 in 2024-03-04T15:59:38.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:37 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T15:59:38.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:37 smithi083 ceph-mon[29452]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:38.956 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-04T15:59:38.956 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-04T15:59:38.956 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-03-04T15:59:38.956 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-03-04T15:59:38.956 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125678 s, 834 MB/s 2024-03-04T15:59:38.956 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-03-04T15:59:39.024 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:38 smithi086 ceph-mon[25584]: osdmap e32: 5 total, 5 up, 5 in 2024-03-04T15:59:39.024 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:38 smithi086 ceph-mon[25584]: Detected new or changed devices on smithi086 2024-03-04T15:59:39.024 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:38 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:39.024 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:38 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:39.024 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:38 smithi086 ceph-mon[25584]: Adjusting osd_memory_target on smithi086 to 17088M 2024-03-04T15:59:39.024 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:38 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:39.024 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:38 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:39.024 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:38 smithi086 ceph-mon[25584]: osdmap e33: 5 total, 5 up, 5 in 2024-03-04T15:59:39.140 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:38 smithi083 ceph-mon[29452]: osdmap e32: 5 total, 5 up, 5 in 2024-03-04T15:59:39.140 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:38 smithi083 ceph-mon[29452]: Detected new or changed devices on smithi086 2024-03-04T15:59:39.140 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:38 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:39.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:38 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:39.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:38 smithi083 ceph-mon[29452]: Adjusting osd_memory_target on smithi086 to 17088M 2024-03-04T15:59:39.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:38 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:39.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:38 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:39.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:38 smithi083 ceph-mon[29452]: osdmap e33: 5 total, 5 up, 5 in 2024-03-04T15:59:39.141 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:38 smithi083 ceph-mon[24738]: osdmap e32: 5 total, 5 up, 5 in 2024-03-04T15:59:39.141 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:38 smithi083 ceph-mon[24738]: Detected new or changed devices on smithi086 2024-03-04T15:59:39.141 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:38 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:39.141 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:38 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:39.141 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:38 smithi083 ceph-mon[24738]: Adjusting osd_memory_target on smithi086 to 17088M 2024-03-04T15:59:39.141 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:38 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:39.141 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:38 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:39.142 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:38 smithi083 ceph-mon[24738]: osdmap e33: 5 total, 5 up, 5 in 2024-03-04T15:59:39.532 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi086:vg_nvme/lv_3 2024-03-04T15:59:40.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:39 smithi086 ceph-mon[25584]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:40.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:39 smithi083 ceph-mon[24738]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:40.153 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:39 smithi083 ceph-mon[29452]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:40.824 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:40 smithi086 ceph-mon[25584]: osdmap e34: 5 total, 5 up, 5 in 2024-03-04T15:59:40.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:40 smithi083 ceph-mon[24738]: osdmap e34: 5 total, 5 up, 5 in 2024-03-04T15:59:40.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:40 smithi083 ceph-mon[29452]: osdmap e34: 5 total, 5 up, 5 in 2024-03-04T15:59:42.112 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:41 smithi086 ceph-mon[25584]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-04T15:59:42.113 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:41 smithi086 ceph-mon[25584]: osdmap e35: 5 total, 5 up, 5 in 2024-03-04T15:59:42.113 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:41 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:59:42.113 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:41 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:59:42.113 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:41 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:42.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:41 smithi083 ceph-mon[24738]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-04T15:59:42.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:41 smithi083 ceph-mon[24738]: osdmap e35: 5 total, 5 up, 5 in 2024-03-04T15:59:42.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:41 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:59:42.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:41 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:59:42.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:41 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:42.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:41 smithi083 ceph-mon[29452]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-04T15:59:42.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:41 smithi083 ceph-mon[29452]: osdmap e35: 5 total, 5 up, 5 in 2024-03-04T15:59:42.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:41 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:59:42.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:41 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:59:42.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:41 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:42.936 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:42 smithi086 ceph-mon[25584]: from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:59:43.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:42 smithi083 ceph-mon[24738]: from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:59:43.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:42 smithi083 ceph-mon[29452]: from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:59:44.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:43 smithi086 ceph-mon[25584]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-04T15:59:44.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:43 smithi086 ceph-mon[25584]: from='client.? 172.21.15.86:0/2638758033' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e1dc2f0-4696-49d9-8691-109139cc3558"}]: dispatch 2024-03-04T15:59:44.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:43 smithi086 ceph-mon[25584]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e1dc2f0-4696-49d9-8691-109139cc3558"}]: dispatch 2024-03-04T15:59:44.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:43 smithi086 ceph-mon[25584]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e1dc2f0-4696-49d9-8691-109139cc3558"}]': finished 2024-03-04T15:59:44.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:43 smithi086 ceph-mon[25584]: osdmap e36: 6 total, 5 up, 6 in 2024-03-04T15:59:44.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:43 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T15:59:44.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:43 smithi083 ceph-mon[24738]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-04T15:59:44.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:43 smithi083 ceph-mon[24738]: from='client.? 172.21.15.86:0/2638758033' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e1dc2f0-4696-49d9-8691-109139cc3558"}]: dispatch 2024-03-04T15:59:44.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:43 smithi083 ceph-mon[24738]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e1dc2f0-4696-49d9-8691-109139cc3558"}]: dispatch 2024-03-04T15:59:44.243 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:43 smithi083 ceph-mon[24738]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e1dc2f0-4696-49d9-8691-109139cc3558"}]': finished 2024-03-04T15:59:44.243 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:43 smithi083 ceph-mon[24738]: osdmap e36: 6 total, 5 up, 6 in 2024-03-04T15:59:44.243 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:43 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T15:59:44.246 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:43 smithi083 ceph-mon[29452]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-04T15:59:44.246 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:43 smithi083 ceph-mon[29452]: from='client.? 172.21.15.86:0/2638758033' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e1dc2f0-4696-49d9-8691-109139cc3558"}]: dispatch 2024-03-04T15:59:44.246 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:43 smithi083 ceph-mon[29452]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e1dc2f0-4696-49d9-8691-109139cc3558"}]: dispatch 2024-03-04T15:59:44.246 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:43 smithi083 ceph-mon[29452]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e1dc2f0-4696-49d9-8691-109139cc3558"}]': finished 2024-03-04T15:59:44.246 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:43 smithi083 ceph-mon[29452]: osdmap e36: 6 total, 5 up, 6 in 2024-03-04T15:59:44.246 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:43 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T15:59:45.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:44 smithi086 ceph-mon[25584]: from='client.? 172.21.15.86:0/1513759314' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:59:45.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:44 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:59:45.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:44 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:59:45.239 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 15:59:44 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T15:59:44.768+0000 7f86f824a700 -1 calc_pg_upmaps abort due to max <= 0 2024-03-04T15:59:45.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:44 smithi083 ceph-mon[24738]: from='client.? 172.21.15.86:0/1513759314' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:59:45.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:44 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:59:45.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:44 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:59:45.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:44 smithi083 ceph-mon[29452]: from='client.? 172.21.15.86:0/1513759314' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:59:45.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:44 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:59:45.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:44 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:59:46.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:45 smithi086 ceph-mon[25584]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-04T15:59:46.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:45 smithi083 ceph-mon[24738]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-04T15:59:46.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:45 smithi083 ceph-mon[29452]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-04T15:59:47.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:47 smithi086 ceph-mon[25584]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:48.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:47 smithi083 ceph-mon[24738]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:48.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:47 smithi083 ceph-mon[29452]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:50.077 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:49 smithi086 ceph-mon[25584]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:50.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:49 smithi083 ceph-mon[24738]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:50.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:49 smithi083 ceph-mon[29452]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:51.369 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:51 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T15:59:51.369 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:51 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:51.369 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:51 smithi083 ceph-mon[29452]: Deploying daemon osd.5 on smithi086 2024-03-04T15:59:51.369 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:51 smithi083 ceph-mon[29452]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:51.370 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:51 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T15:59:51.370 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:51 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:51.370 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:51 smithi083 ceph-mon[24738]: Deploying daemon osd.5 on smithi086 2024-03-04T15:59:51.370 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:51 smithi083 ceph-mon[24738]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:51.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:51 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T15:59:51.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:51 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:51.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:51 smithi086 ceph-mon[25584]: Deploying daemon osd.5 on smithi086 2024-03-04T15:59:51.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:51 smithi086 ceph-mon[25584]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:54.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:53 smithi086 ceph-mon[25584]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:54.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:53 smithi083 ceph-mon[24738]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:54.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:53 smithi083 ceph-mon[29452]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:56.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:55 smithi086 ceph-mon[25584]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:56.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:55 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:56.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:55 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:56.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:55 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:56.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:55 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:56.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:55 smithi083 ceph-mon[24738]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:56.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:55 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:56.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:55 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:56.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:55 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:56.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:55 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:56.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:55 smithi083 ceph-mon[29452]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:56.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:55 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:56.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:55 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:56.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:55 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:56.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:55 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:57.863 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:57 smithi086 ceph-mon[25584]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:58.167 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 5 on host 'smithi086' 2024-03-04T15:59:58.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:57 smithi083 ceph-mon[24738]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:58.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:57 smithi083 ceph-mon[29452]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:58.730 DEBUG:teuthology.orchestra.run.smithi086:osd.5> sudo journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.5.service 2024-03-04T15:59:58.732 INFO:tasks.cephadm:Deploying osd.6 on smithi086 with /dev/vg_nvme/lv_2... 2024-03-04T15:59:58.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-04T15:59:59.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:59 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:59.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:59 smithi086 ceph-mon[25584]: from='osd.5 [v2:172.21.15.86:6808/3306461240,v1:172.21.15.86:6809/3306461240]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T15:59:59.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:59 smithi086 ceph-mon[25584]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T15:59:59.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:59 smithi086 ceph-mon[25584]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:59.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 15:59:59 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:59.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:59 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:59.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:59 smithi083 ceph-mon[24738]: from='osd.5 [v2:172.21.15.86:6808/3306461240,v1:172.21.15.86:6809/3306461240]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T15:59:59.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:59 smithi083 ceph-mon[24738]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T15:59:59.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:59 smithi083 ceph-mon[24738]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:59.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 15:59:59 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:59.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:59 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T15:59:59.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:59 smithi083 ceph-mon[29452]: from='osd.5 [v2:172.21.15.86:6808/3306461240,v1:172.21.15.86:6809/3306461240]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T15:59:59.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:59 smithi083 ceph-mon[29452]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T15:59:59.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:59 smithi083 ceph-mon[29452]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T15:59:59.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 15:59:59 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:00.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:00 smithi086 ceph-mon[25584]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-04T16:00:00.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:00 smithi086 ceph-mon[25584]: osdmap e37: 6 total, 5 up, 6 in 2024-03-04T16:00:00.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:00 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:00:00.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:00 smithi086 ceph-mon[25584]: from='osd.5 [v2:172.21.15.86:6808/3306461240,v1:172.21.15.86:6809/3306461240]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:00:00.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:00 smithi086 ceph-mon[25584]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:00:00.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:00 smithi086 ceph-mon[25584]: overall HEALTH_OK 2024-03-04T16:00:00.235 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:00:00 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[31802]: 2024-03-04T16:00:00.186+0000 7fe9458f2700 -1 osd.5 0 waiting for initial osdmap 2024-03-04T16:00:00.235 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:00:00 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[31802]: 2024-03-04T16:00:00.191+0000 7fe94228d700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:00:00.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:00 smithi083 ceph-mon[24738]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-04T16:00:00.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:00 smithi083 ceph-mon[24738]: osdmap e37: 6 total, 5 up, 6 in 2024-03-04T16:00:00.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:00 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:00:00.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:00 smithi083 ceph-mon[24738]: from='osd.5 [v2:172.21.15.86:6808/3306461240,v1:172.21.15.86:6809/3306461240]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:00:00.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:00 smithi083 ceph-mon[24738]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:00:00.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:00 smithi083 ceph-mon[24738]: overall HEALTH_OK 2024-03-04T16:00:00.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:00 smithi083 ceph-mon[29452]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-04T16:00:00.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:00 smithi083 ceph-mon[29452]: osdmap e37: 6 total, 5 up, 6 in 2024-03-04T16:00:00.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:00 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:00:00.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:00 smithi083 ceph-mon[29452]: from='osd.5 [v2:172.21.15.86:6808/3306461240,v1:172.21.15.86:6809/3306461240]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:00:00.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:00 smithi083 ceph-mon[29452]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:00:00.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:00 smithi083 ceph-mon[29452]: overall HEALTH_OK 2024-03-04T16:00:01.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:01 smithi086 ceph-mon[25584]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T16:00:01.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:01 smithi086 ceph-mon[25584]: osdmap e38: 6 total, 5 up, 6 in 2024-03-04T16:00:01.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:01 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:00:01.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:01 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:00:01.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:01 smithi086 ceph-mon[25584]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:00:01.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:01 smithi083 ceph-mon[29452]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T16:00:01.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:01 smithi083 ceph-mon[29452]: osdmap e38: 6 total, 5 up, 6 in 2024-03-04T16:00:01.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:01 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:00:01.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:01 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:00:01.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:01 smithi083 ceph-mon[29452]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:00:01.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:01 smithi083 ceph-mon[24738]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T16:00:01.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:01 smithi083 ceph-mon[24738]: osdmap e38: 6 total, 5 up, 6 in 2024-03-04T16:00:01.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:01 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:00:01.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:01 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:00:01.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:01 smithi083 ceph-mon[24738]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:00:01.712 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-04T16:00:01.712 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-04T16:00:01.712 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-03-04T16:00:01.712 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-03-04T16:00:01.712 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012461 s, 841 MB/s 2024-03-04T16:00:01.712 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-03-04T16:00:02.448 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:02 smithi086 ceph-mon[25584]: purged_snaps scrub starts 2024-03-04T16:00:02.449 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:02 smithi086 ceph-mon[25584]: purged_snaps scrub ok 2024-03-04T16:00:02.449 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:02 smithi086 ceph-mon[25584]: osd.5 [v2:172.21.15.86:6808/3306461240,v1:172.21.15.86:6809/3306461240] boot 2024-03-04T16:00:02.449 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:02 smithi086 ceph-mon[25584]: osdmap e39: 6 total, 6 up, 6 in 2024-03-04T16:00:02.449 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:02 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:00:02.449 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:02 smithi086 ceph-mon[25584]: osdmap e40: 6 total, 6 up, 6 in 2024-03-04T16:00:02.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:02 smithi083 ceph-mon[24738]: purged_snaps scrub starts 2024-03-04T16:00:02.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:02 smithi083 ceph-mon[24738]: purged_snaps scrub ok 2024-03-04T16:00:02.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:02 smithi083 ceph-mon[24738]: osd.5 [v2:172.21.15.86:6808/3306461240,v1:172.21.15.86:6809/3306461240] boot 2024-03-04T16:00:02.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:02 smithi083 ceph-mon[24738]: osdmap e39: 6 total, 6 up, 6 in 2024-03-04T16:00:02.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:02 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:00:02.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:02 smithi083 ceph-mon[24738]: osdmap e40: 6 total, 6 up, 6 in 2024-03-04T16:00:02.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:02 smithi083 ceph-mon[29452]: purged_snaps scrub starts 2024-03-04T16:00:02.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:02 smithi083 ceph-mon[29452]: purged_snaps scrub ok 2024-03-04T16:00:02.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:02 smithi083 ceph-mon[29452]: osd.5 [v2:172.21.15.86:6808/3306461240,v1:172.21.15.86:6809/3306461240] boot 2024-03-04T16:00:02.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:02 smithi083 ceph-mon[29452]: osdmap e39: 6 total, 6 up, 6 in 2024-03-04T16:00:02.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:02 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:00:02.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:02 smithi083 ceph-mon[29452]: osdmap e40: 6 total, 6 up, 6 in 2024-03-04T16:00:02.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi086:vg_nvme/lv_2 2024-03-04T16:00:03.251 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:03 smithi086 ceph-mon[25584]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:03.252 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:03 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:03.252 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:03 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:03.252 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:03 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:03.252 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:03 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:03.252 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:03 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:03.252 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:03 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:03.252 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:03 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:03.252 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:03 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:03.252 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:03 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:03.252 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:03 smithi086 ceph-mon[25584]: osdmap e41: 6 total, 6 up, 6 in 2024-03-04T16:00:03.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[29452]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:03.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:03.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:03.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:03.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:03.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:03.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:03.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:03.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:03.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:03.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[29452]: osdmap e41: 6 total, 6 up, 6 in 2024-03-04T16:00:03.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[24738]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:03.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:03.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:03.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:03.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:03.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:03.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:03.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:03.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:03.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:03.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:03 smithi083 ceph-mon[24738]: osdmap e41: 6 total, 6 up, 6 in 2024-03-04T16:00:04.355 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:04 smithi086 ceph-mon[25584]: Detected new or changed devices on smithi086 2024-03-04T16:00:04.355 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:04 smithi086 ceph-mon[25584]: Adjusting osd_memory_target on smithi086 to 8544M 2024-03-04T16:00:04.355 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:04 smithi086 ceph-mon[25584]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-04T16:00:04.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:04 smithi083 ceph-mon[24738]: Detected new or changed devices on smithi086 2024-03-04T16:00:04.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:04 smithi083 ceph-mon[24738]: Adjusting osd_memory_target on smithi086 to 8544M 2024-03-04T16:00:04.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:04 smithi083 ceph-mon[24738]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-04T16:00:04.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:04 smithi083 ceph-mon[29452]: Detected new or changed devices on smithi086 2024-03-04T16:00:04.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:04 smithi083 ceph-mon[29452]: Adjusting osd_memory_target on smithi086 to 8544M 2024-03-04T16:00:04.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:04 smithi083 ceph-mon[29452]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-04T16:00:05.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:05 smithi086 ceph-mon[25584]: osdmap e42: 6 total, 6 up, 6 in 2024-03-04T16:00:05.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:05 smithi086 ceph-mon[25584]: pgmap v132: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:05.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:05 smithi086 ceph-mon[25584]: from='client.34241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:05.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:05 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:00:05.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:05 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:00:05.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:05 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:05.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:05 smithi083 ceph-mon[24738]: osdmap e42: 6 total, 6 up, 6 in 2024-03-04T16:00:05.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:05 smithi083 ceph-mon[24738]: pgmap v132: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:05.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:05 smithi083 ceph-mon[24738]: from='client.34241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:05.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:05 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:00:05.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:05 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:00:05.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:05 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:05.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:05 smithi083 ceph-mon[29452]: osdmap e42: 6 total, 6 up, 6 in 2024-03-04T16:00:05.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:05 smithi083 ceph-mon[29452]: pgmap v132: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:05.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:05 smithi083 ceph-mon[29452]: from='client.34241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:05.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:05 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:00:05.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:05 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:00:05.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:05 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:06.376 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:06 smithi086 ceph-mon[25584]: osdmap e43: 6 total, 6 up, 6 in 2024-03-04T16:00:06.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:06 smithi083 ceph-mon[24738]: osdmap e43: 6 total, 6 up, 6 in 2024-03-04T16:00:06.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:06 smithi083 ceph-mon[29452]: osdmap e43: 6 total, 6 up, 6 in 2024-03-04T16:00:07.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:07 smithi086 ceph-mon[25584]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-04T16:00:07.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:07 smithi086 ceph-mon[25584]: from='client.? 172.21.15.86:0/2460513067' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd5216b0-8187-40f2-a10f-724f8f8c95a1"}]: dispatch 2024-03-04T16:00:07.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:07 smithi086 ceph-mon[25584]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd5216b0-8187-40f2-a10f-724f8f8c95a1"}]: dispatch 2024-03-04T16:00:07.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:07 smithi086 ceph-mon[25584]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd5216b0-8187-40f2-a10f-724f8f8c95a1"}]': finished 2024-03-04T16:00:07.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:07 smithi086 ceph-mon[25584]: osdmap e44: 7 total, 6 up, 7 in 2024-03-04T16:00:07.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:07 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:00:07.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:07 smithi083 ceph-mon[29452]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-04T16:00:07.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:07 smithi083 ceph-mon[29452]: from='client.? 172.21.15.86:0/2460513067' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd5216b0-8187-40f2-a10f-724f8f8c95a1"}]: dispatch 2024-03-04T16:00:07.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:07 smithi083 ceph-mon[29452]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd5216b0-8187-40f2-a10f-724f8f8c95a1"}]: dispatch 2024-03-04T16:00:07.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:07 smithi083 ceph-mon[29452]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd5216b0-8187-40f2-a10f-724f8f8c95a1"}]': finished 2024-03-04T16:00:07.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:07 smithi083 ceph-mon[29452]: osdmap e44: 7 total, 6 up, 7 in 2024-03-04T16:00:07.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:07 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:00:07.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:07 smithi083 ceph-mon[24738]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-04T16:00:07.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:07 smithi083 ceph-mon[24738]: from='client.? 172.21.15.86:0/2460513067' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd5216b0-8187-40f2-a10f-724f8f8c95a1"}]: dispatch 2024-03-04T16:00:07.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:07 smithi083 ceph-mon[24738]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd5216b0-8187-40f2-a10f-724f8f8c95a1"}]: dispatch 2024-03-04T16:00:07.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:07 smithi083 ceph-mon[24738]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd5216b0-8187-40f2-a10f-724f8f8c95a1"}]': finished 2024-03-04T16:00:07.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:07 smithi083 ceph-mon[24738]: osdmap e44: 7 total, 6 up, 7 in 2024-03-04T16:00:07.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:07 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:00:08.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:08 smithi086 ceph-mon[25584]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-04T16:00:08.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:08 smithi086 ceph-mon[25584]: Cluster is now healthy 2024-03-04T16:00:08.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:08 smithi086 ceph-mon[25584]: from='client.? 172.21.15.86:0/3131279913' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:00:08.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:08 smithi083 ceph-mon[24738]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-04T16:00:08.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:08 smithi083 ceph-mon[24738]: Cluster is now healthy 2024-03-04T16:00:08.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:08 smithi083 ceph-mon[24738]: from='client.? 172.21.15.86:0/3131279913' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:00:08.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:08 smithi083 ceph-mon[29452]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-04T16:00:08.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:08 smithi083 ceph-mon[29452]: Cluster is now healthy 2024-03-04T16:00:08.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:08 smithi083 ceph-mon[29452]: from='client.? 172.21.15.86:0/3131279913' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:00:09.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:09 smithi086 ceph-mon[25584]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-04T16:00:09.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:09 smithi083 ceph-mon[24738]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-04T16:00:09.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:09 smithi083 ceph-mon[29452]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-04T16:00:11.215 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:11 smithi086 ceph-mon[25584]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-04T16:00:11.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:11 smithi083 ceph-mon[24738]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-04T16:00:11.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:11 smithi083 ceph-mon[29452]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-04T16:00:13.910 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:13 smithi086 ceph-mon[25584]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-04T16:00:13.910 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:13 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-04T16:00:13.910 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:13 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:14.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:13 smithi083 ceph-mon[29452]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-04T16:00:14.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:13 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-04T16:00:14.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:13 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:14.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:13 smithi083 ceph-mon[24738]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-04T16:00:14.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:13 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-04T16:00:14.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:13 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:15.095 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:14 smithi086 ceph-mon[25584]: Deploying daemon osd.6 on smithi086 2024-03-04T16:00:15.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:14 smithi083 ceph-mon[29452]: Deploying daemon osd.6 on smithi086 2024-03-04T16:00:15.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:14 smithi083 ceph-mon[24738]: Deploying daemon osd.6 on smithi086 2024-03-04T16:00:15.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:15 smithi086 ceph-mon[25584]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-04T16:00:16.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:15 smithi083 ceph-mon[29452]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-04T16:00:16.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:15 smithi083 ceph-mon[24738]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-04T16:00:18.056 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:17 smithi086 ceph-mon[25584]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:18.105 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:17 smithi083 ceph-mon[24738]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:18.106 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:17 smithi083 ceph-mon[29452]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:19.346 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:19 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:19.346 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:19 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:19.346 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:19 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:19.346 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:19 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:19.346 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:19 smithi086 ceph-mon[25584]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:19.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:19 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:19.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:19 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:19.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:19 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:19.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:19 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:19.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:19 smithi083 ceph-mon[24738]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:19.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:19 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:19.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:19 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:19.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:19 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:19.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:19 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:19.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:19 smithi083 ceph-mon[29452]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:21.356 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 6 on host 'smithi086' 2024-03-04T16:00:21.616 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:21 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:21.616 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:21 smithi086 ceph-mon[25584]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:21.617 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:21 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:21.617 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:21 smithi086 ceph-mon[25584]: from='osd.6 [v2:172.21.15.86:6816/3999097134,v1:172.21.15.86:6817/3999097134]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T16:00:21.617 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:21 smithi086 ceph-mon[25584]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T16:00:21.617 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:21 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:21.617 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:21 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:21.617 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:21 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:21.617 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:21 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:21.965 DEBUG:teuthology.orchestra.run.smithi086:osd.6> sudo journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.6.service 2024-03-04T16:00:21.967 INFO:tasks.cephadm:Deploying osd.7 on smithi086 with /dev/vg_nvme/lv_1... 2024-03-04T16:00:21.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-04T16:00:21.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:21 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:21.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:21 smithi083 ceph-mon[24738]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:21.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:21 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:21.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:21 smithi083 ceph-mon[24738]: from='osd.6 [v2:172.21.15.86:6816/3999097134,v1:172.21.15.86:6817/3999097134]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T16:00:21.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:21 smithi083 ceph-mon[24738]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T16:00:21.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:21 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:21.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:21 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:21.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:21 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:21.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:21 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:21.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:21 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:21.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:21 smithi083 ceph-mon[29452]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:21.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:21 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:21.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:21 smithi083 ceph-mon[29452]: from='osd.6 [v2:172.21.15.86:6816/3999097134,v1:172.21.15.86:6817/3999097134]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T16:00:21.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:21 smithi083 ceph-mon[29452]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T16:00:21.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:21 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:21.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:21 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:21.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:21 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:21.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:21 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:23.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:23 smithi086 ceph-mon[25584]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-04T16:00:23.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:23 smithi086 ceph-mon[25584]: osdmap e45: 7 total, 6 up, 7 in 2024-03-04T16:00:23.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:23 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:00:23.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:23 smithi086 ceph-mon[25584]: from='osd.6 [v2:172.21.15.86:6816/3999097134,v1:172.21.15.86:6817/3999097134]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:00:23.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:23 smithi086 ceph-mon[25584]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:00:23.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:23 smithi086 ceph-mon[25584]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:23.485 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:00:23 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[34681]: 2024-03-04T16:00:23.068+0000 7fcf9b3be700 -1 osd.6 0 waiting for initial osdmap 2024-03-04T16:00:23.486 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:00:23 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[34681]: 2024-03-04T16:00:23.076+0000 7fcf95d55700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:00:23.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:23 smithi083 ceph-mon[29452]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-04T16:00:23.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:23 smithi083 ceph-mon[29452]: osdmap e45: 7 total, 6 up, 7 in 2024-03-04T16:00:23.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:23 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:00:23.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:23 smithi083 ceph-mon[29452]: from='osd.6 [v2:172.21.15.86:6816/3999097134,v1:172.21.15.86:6817/3999097134]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:00:23.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:23 smithi083 ceph-mon[29452]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:00:23.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:23 smithi083 ceph-mon[29452]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:23.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:23 smithi083 ceph-mon[24738]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-04T16:00:23.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:23 smithi083 ceph-mon[24738]: osdmap e45: 7 total, 6 up, 7 in 2024-03-04T16:00:23.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:23 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:00:23.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:23 smithi083 ceph-mon[24738]: from='osd.6 [v2:172.21.15.86:6816/3999097134,v1:172.21.15.86:6817/3999097134]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:00:23.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:23 smithi083 ceph-mon[24738]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:00:23.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:23 smithi083 ceph-mon[24738]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:00:24.342 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:24 smithi086 ceph-mon[25584]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T16:00:24.342 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:24 smithi086 ceph-mon[25584]: osdmap e46: 7 total, 6 up, 7 in 2024-03-04T16:00:24.342 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:24 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:00:24.342 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:24 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:00:24.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:24 smithi083 ceph-mon[24738]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T16:00:24.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:24 smithi083 ceph-mon[24738]: osdmap e46: 7 total, 6 up, 7 in 2024-03-04T16:00:24.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:24 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:00:24.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:24 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:00:24.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:24 smithi083 ceph-mon[29452]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T16:00:24.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:24 smithi083 ceph-mon[29452]: osdmap e46: 7 total, 6 up, 7 in 2024-03-04T16:00:24.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:24 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:00:24.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:24 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:00:24.822 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-04T16:00:24.822 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-04T16:00:24.822 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-03-04T16:00:24.822 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-03-04T16:00:24.822 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127756 s, 821 MB/s 2024-03-04T16:00:24.822 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-03-04T16:00:25.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:25 smithi086 ceph-mon[25584]: purged_snaps scrub starts 2024-03-04T16:00:25.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:25 smithi086 ceph-mon[25584]: purged_snaps scrub ok 2024-03-04T16:00:25.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:25 smithi086 ceph-mon[25584]: osd.6 [v2:172.21.15.86:6816/3999097134,v1:172.21.15.86:6817/3999097134] boot 2024-03-04T16:00:25.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:25 smithi086 ceph-mon[25584]: osdmap e47: 7 total, 7 up, 7 in 2024-03-04T16:00:25.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:25 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:00:25.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:25 smithi086 ceph-mon[25584]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:00:25.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:25 smithi083 ceph-mon[24738]: purged_snaps scrub starts 2024-03-04T16:00:25.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:25 smithi083 ceph-mon[24738]: purged_snaps scrub ok 2024-03-04T16:00:25.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:25 smithi083 ceph-mon[24738]: osd.6 [v2:172.21.15.86:6816/3999097134,v1:172.21.15.86:6817/3999097134] boot 2024-03-04T16:00:25.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:25 smithi083 ceph-mon[24738]: osdmap e47: 7 total, 7 up, 7 in 2024-03-04T16:00:25.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:25 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:00:25.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:25 smithi083 ceph-mon[24738]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:00:25.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:25 smithi083 ceph-mon[29452]: purged_snaps scrub starts 2024-03-04T16:00:25.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:25 smithi083 ceph-mon[29452]: purged_snaps scrub ok 2024-03-04T16:00:25.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:25 smithi083 ceph-mon[29452]: osd.6 [v2:172.21.15.86:6816/3999097134,v1:172.21.15.86:6817/3999097134] boot 2024-03-04T16:00:25.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:25 smithi083 ceph-mon[29452]: osdmap e47: 7 total, 7 up, 7 in 2024-03-04T16:00:25.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:25 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:00:25.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:25 smithi083 ceph-mon[29452]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:00:25.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi086:vg_nvme/lv_1 2024-03-04T16:00:26.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:26 smithi086 ceph-mon[25584]: osdmap e48: 7 total, 7 up, 7 in 2024-03-04T16:00:26.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:26 smithi086 ceph-mon[25584]: osdmap e49: 7 total, 7 up, 7 in 2024-03-04T16:00:26.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:26 smithi083 ceph-mon[24738]: osdmap e48: 7 total, 7 up, 7 in 2024-03-04T16:00:26.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:26 smithi083 ceph-mon[24738]: osdmap e49: 7 total, 7 up, 7 in 2024-03-04T16:00:26.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:26 smithi083 ceph-mon[29452]: osdmap e48: 7 total, 7 up, 7 in 2024-03-04T16:00:26.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:26 smithi083 ceph-mon[29452]: osdmap e49: 7 total, 7 up, 7 in 2024-03-04T16:00:27.434 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:27 smithi086 ceph-mon[25584]: Detected new or changed devices on smithi086 2024-03-04T16:00:27.434 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:27 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:27.434 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:27 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:27.434 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:27 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:27.435 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:27 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:27.435 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:27 smithi086 ceph-mon[25584]: Adjusting osd_memory_target on smithi086 to 5696M 2024-03-04T16:00:27.435 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:27 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:27.435 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:27 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:27.435 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:27 smithi086 ceph-mon[25584]: pgmap v150: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 1/6 objects misplaced (16.667%); 49 KiB/s, 0 objects/s recovering 2024-03-04T16:00:27.435 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:27 smithi086 ceph-mon[25584]: osdmap e50: 7 total, 7 up, 7 in 2024-03-04T16:00:27.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[24738]: Detected new or changed devices on smithi086 2024-03-04T16:00:27.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:27.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:27.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:27.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:27.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[24738]: Adjusting osd_memory_target on smithi086 to 5696M 2024-03-04T16:00:27.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:27.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:27.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[24738]: pgmap v150: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 1/6 objects misplaced (16.667%); 49 KiB/s, 0 objects/s recovering 2024-03-04T16:00:27.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[24738]: osdmap e50: 7 total, 7 up, 7 in 2024-03-04T16:00:27.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[29452]: Detected new or changed devices on smithi086 2024-03-04T16:00:27.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:27.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:27.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:27.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:27.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[29452]: Adjusting osd_memory_target on smithi086 to 5696M 2024-03-04T16:00:27.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:27.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:27.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[29452]: pgmap v150: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 1/6 objects misplaced (16.667%); 49 KiB/s, 0 objects/s recovering 2024-03-04T16:00:27.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:27 smithi083 ceph-mon[29452]: osdmap e50: 7 total, 7 up, 7 in 2024-03-04T16:00:29.401 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:29 smithi086 ceph-mon[25584]: osdmap e51: 7 total, 7 up, 7 in 2024-03-04T16:00:29.402 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:29 smithi086 ceph-mon[25584]: from='client.34268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:29.402 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:29 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:00:29.402 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:29 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:00:29.402 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:29 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:29.402 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:29 smithi086 ceph-mon[25584]: pgmap v153: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 1/6 objects misplaced (16.667%); 49 KiB/s, 0 objects/s recovering 2024-03-04T16:00:29.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:29 smithi083 ceph-mon[24738]: osdmap e51: 7 total, 7 up, 7 in 2024-03-04T16:00:29.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:29 smithi083 ceph-mon[24738]: from='client.34268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:29.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:29 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:00:29.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:29 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:00:29.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:29 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:29.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:29 smithi083 ceph-mon[24738]: pgmap v153: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 1/6 objects misplaced (16.667%); 49 KiB/s, 0 objects/s recovering 2024-03-04T16:00:29.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:29 smithi083 ceph-mon[29452]: osdmap e51: 7 total, 7 up, 7 in 2024-03-04T16:00:29.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:29 smithi083 ceph-mon[29452]: from='client.34268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:29.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:29 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:00:29.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:29 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:00:29.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:29 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:29.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:29 smithi083 ceph-mon[29452]: pgmap v153: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 1/6 objects misplaced (16.667%); 49 KiB/s, 0 objects/s recovering 2024-03-04T16:00:31.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:31 smithi086 ceph-mon[25584]: from='client.? 172.21.15.86:0/3480930504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11856cf1-448a-4a87-822a-4df57524fa47"}]: dispatch 2024-03-04T16:00:31.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:31 smithi086 ceph-mon[25584]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11856cf1-448a-4a87-822a-4df57524fa47"}]: dispatch 2024-03-04T16:00:31.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:31 smithi086 ceph-mon[25584]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11856cf1-448a-4a87-822a-4df57524fa47"}]': finished 2024-03-04T16:00:31.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:31 smithi086 ceph-mon[25584]: osdmap e52: 8 total, 7 up, 8 in 2024-03-04T16:00:31.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:31 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:00:31.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:31 smithi086 ceph-mon[25584]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-04T16:00:31.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:31 smithi086 ceph-mon[25584]: from='client.? 172.21.15.86:0/182285256' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:00:31.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:31 smithi083 ceph-mon[24738]: from='client.? 172.21.15.86:0/3480930504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11856cf1-448a-4a87-822a-4df57524fa47"}]: dispatch 2024-03-04T16:00:31.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:31 smithi083 ceph-mon[24738]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11856cf1-448a-4a87-822a-4df57524fa47"}]: dispatch 2024-03-04T16:00:31.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:31 smithi083 ceph-mon[24738]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11856cf1-448a-4a87-822a-4df57524fa47"}]': finished 2024-03-04T16:00:31.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:31 smithi083 ceph-mon[24738]: osdmap e52: 8 total, 7 up, 8 in 2024-03-04T16:00:31.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:31 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:00:31.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:31 smithi083 ceph-mon[24738]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-04T16:00:31.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:31 smithi083 ceph-mon[24738]: from='client.? 172.21.15.86:0/182285256' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:00:31.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:31 smithi083 ceph-mon[29452]: from='client.? 172.21.15.86:0/3480930504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11856cf1-448a-4a87-822a-4df57524fa47"}]: dispatch 2024-03-04T16:00:31.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:31 smithi083 ceph-mon[29452]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11856cf1-448a-4a87-822a-4df57524fa47"}]: dispatch 2024-03-04T16:00:31.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:31 smithi083 ceph-mon[29452]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11856cf1-448a-4a87-822a-4df57524fa47"}]': finished 2024-03-04T16:00:31.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:31 smithi083 ceph-mon[29452]: osdmap e52: 8 total, 7 up, 8 in 2024-03-04T16:00:31.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:31 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:00:31.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:31 smithi083 ceph-mon[29452]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-04T16:00:31.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:31 smithi083 ceph-mon[29452]: from='client.? 172.21.15.86:0/182285256' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:00:34.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:33 smithi086 ceph-mon[25584]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-03-04T16:00:34.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:33 smithi083 ceph-mon[24738]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-03-04T16:00:34.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:33 smithi083 ceph-mon[29452]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-03-04T16:00:36.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:35 smithi086 ceph-mon[25584]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 42 KiB/s, 0 objects/s recovering 2024-03-04T16:00:36.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:35 smithi083 ceph-mon[24738]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 42 KiB/s, 0 objects/s recovering 2024-03-04T16:00:36.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:35 smithi083 ceph-mon[29452]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 42 KiB/s, 0 objects/s recovering 2024-03-04T16:00:36.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:36 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-04T16:00:36.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:36 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:37.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:36 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-04T16:00:37.344 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:36 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:37.345 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:36 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-04T16:00:37.345 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:36 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:37.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:37 smithi086 ceph-mon[25584]: Deploying daemon osd.7 on smithi086 2024-03-04T16:00:37.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:37 smithi086 ceph-mon[25584]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 37 KiB/s, 0 objects/s recovering 2024-03-04T16:00:38.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:37 smithi083 ceph-mon[24738]: Deploying daemon osd.7 on smithi086 2024-03-04T16:00:38.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:37 smithi083 ceph-mon[24738]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 37 KiB/s, 0 objects/s recovering 2024-03-04T16:00:38.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:37 smithi083 ceph-mon[29452]: Deploying daemon osd.7 on smithi086 2024-03-04T16:00:38.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:37 smithi083 ceph-mon[29452]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 37 KiB/s, 0 objects/s recovering 2024-03-04T16:00:40.081 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:39 smithi086 ceph-mon[25584]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 32 KiB/s, 0 objects/s recovering 2024-03-04T16:00:40.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:39 smithi083 ceph-mon[24738]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 32 KiB/s, 0 objects/s recovering 2024-03-04T16:00:40.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:39 smithi083 ceph-mon[29452]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 32 KiB/s, 0 objects/s recovering 2024-03-04T16:00:41.344 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:41 smithi086 ceph-mon[25584]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:00:41.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:41 smithi083 ceph-mon[24738]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:00:41.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:41 smithi083 ceph-mon[29452]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:00:42.956 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:42 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:42.956 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:42 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:42.956 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:42 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:42.956 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:42 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:42.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:42 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:42.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:42 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:42.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:42 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:42.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:42 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:42.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:42 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:42.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:42 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:42.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:42 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:42.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:42 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:43.943 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:43 smithi086 ceph-mon[25584]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:00:43.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:43 smithi083 ceph-mon[24738]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:00:43.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:43 smithi083 ceph-mon[29452]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:00:45.043 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 7 on host 'smithi086' 2024-03-04T16:00:45.358 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:45 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:45.358 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:45 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:45.358 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:45 smithi086 ceph-mon[25584]: from='osd.7 [v2:172.21.15.86:6824/702851952,v1:172.21.15.86:6825/702851952]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T16:00:45.358 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:45 smithi086 ceph-mon[25584]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T16:00:45.358 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:45 smithi086 ceph-mon[25584]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:00:45.358 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:45 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:00:45.358 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:45 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:00:45.358 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:45 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:45.358 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:45 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:45.359 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:45 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:45.359 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:45 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:45.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:45.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:45.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[24738]: from='osd.7 [v2:172.21.15.86:6824/702851952,v1:172.21.15.86:6825/702851952]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T16:00:45.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[24738]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T16:00:45.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[24738]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:00:45.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:00:45.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:00:45.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:45.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:45.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:45.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:45.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:45.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:45.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[29452]: from='osd.7 [v2:172.21.15.86:6824/702851952,v1:172.21.15.86:6825/702851952]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T16:00:45.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[29452]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T16:00:45.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[29452]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:00:45.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:00:45.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:00:45.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:45.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:45.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:45.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:45 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:45.647 DEBUG:teuthology.orchestra.run.smithi086:osd.7> sudo journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.7.service 2024-03-04T16:00:45.649 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-04T16:00:45.649 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph osd stat -f json 2024-03-04T16:00:46.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:46 smithi083 ceph-mon[24738]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-04T16:00:46.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:46 smithi083 ceph-mon[24738]: osdmap e53: 8 total, 7 up, 8 in 2024-03-04T16:00:46.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:46 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:00:46.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:46 smithi083 ceph-mon[24738]: from='osd.7 [v2:172.21.15.86:6824/702851952,v1:172.21.15.86:6825/702851952]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:00:46.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:46 smithi083 ceph-mon[24738]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:00:46.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:46 smithi083 ceph-mon[29452]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-04T16:00:46.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:46 smithi083 ceph-mon[29452]: osdmap e53: 8 total, 7 up, 8 in 2024-03-04T16:00:46.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:46 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:00:46.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:46 smithi083 ceph-mon[29452]: from='osd.7 [v2:172.21.15.86:6824/702851952,v1:172.21.15.86:6825/702851952]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:00:46.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:46 smithi083 ceph-mon[29452]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:00:46.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:46 smithi086 ceph-mon[25584]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-04T16:00:46.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:46 smithi086 ceph-mon[25584]: osdmap e53: 8 total, 7 up, 8 in 2024-03-04T16:00:46.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:46 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:00:46.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:46 smithi086 ceph-mon[25584]: from='osd.7 [v2:172.21.15.86:6824/702851952,v1:172.21.15.86:6825/702851952]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:00:46.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:46 smithi086 ceph-mon[25584]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:00:46.486 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:00:46 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[37676]: 2024-03-04T16:00:46.206+0000 7fc4807e0700 -1 osd.7 0 waiting for initial osdmap 2024-03-04T16:00:46.486 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:00:46 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[37676]: 2024-03-04T16:00:46.213+0000 7fc47a175700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:00:47.281 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:47 smithi083 ceph-mon[24738]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T16:00:47.282 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:47 smithi083 ceph-mon[24738]: osdmap e54: 8 total, 7 up, 8 in 2024-03-04T16:00:47.282 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:47 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:00:47.282 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:47 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:00:47.282 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:47 smithi083 ceph-mon[24738]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:00:47.282 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:47 smithi083 ceph-mon[29452]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T16:00:47.282 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:47 smithi083 ceph-mon[29452]: osdmap e54: 8 total, 7 up, 8 in 2024-03-04T16:00:47.282 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:47 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:00:47.283 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:47 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:00:47.283 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:47 smithi083 ceph-mon[29452]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:00:47.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:47 smithi086 ceph-mon[25584]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T16:00:47.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:47 smithi086 ceph-mon[25584]: osdmap e54: 8 total, 7 up, 8 in 2024-03-04T16:00:47.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:47 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:00:47.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:47 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:00:47.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:47 smithi086 ceph-mon[25584]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:00:47.786 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-04T16:00:48.454 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:48 smithi086 ceph-mon[25584]: purged_snaps scrub starts 2024-03-04T16:00:48.455 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:48 smithi086 ceph-mon[25584]: purged_snaps scrub ok 2024-03-04T16:00:48.455 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:48 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:00:48.455 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:48 smithi086 ceph-mon[25584]: osd.7 [v2:172.21.15.86:6824/702851952,v1:172.21.15.86:6825/702851952] boot 2024-03-04T16:00:48.455 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:48 smithi086 ceph-mon[25584]: osdmap e55: 8 total, 8 up, 8 in 2024-03-04T16:00:48.455 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:48 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:00:48.455 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:48 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/214939586' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T16:00:48.480 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:48 smithi083 ceph-mon[24738]: purged_snaps scrub starts 2024-03-04T16:00:48.480 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:48 smithi083 ceph-mon[24738]: purged_snaps scrub ok 2024-03-04T16:00:48.480 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:48 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:00:48.480 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:48 smithi083 ceph-mon[24738]: osd.7 [v2:172.21.15.86:6824/702851952,v1:172.21.15.86:6825/702851952] boot 2024-03-04T16:00:48.480 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:48 smithi083 ceph-mon[24738]: osdmap e55: 8 total, 8 up, 8 in 2024-03-04T16:00:48.480 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:48 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:00:48.481 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:48 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/214939586' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T16:00:48.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:48 smithi083 ceph-mon[29452]: purged_snaps scrub starts 2024-03-04T16:00:48.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:48 smithi083 ceph-mon[29452]: purged_snaps scrub ok 2024-03-04T16:00:48.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:48 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:00:48.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:48 smithi083 ceph-mon[29452]: osd.7 [v2:172.21.15.86:6824/702851952,v1:172.21.15.86:6825/702851952] boot 2024-03-04T16:00:48.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:48 smithi083 ceph-mon[29452]: osdmap e55: 8 total, 8 up, 8 in 2024-03-04T16:00:48.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:48 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:00:48.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:48 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/214939586' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T16:00:48.509 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1709568047,"num_in_osds":8,"osd_in_since":1709568030,"num_remapped_pgs":1} 2024-03-04T16:00:48.510 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T16:00:48.732 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:00:49.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:49 smithi086 ceph-mon[25584]: osdmap e56: 8 total, 8 up, 8 in 2024-03-04T16:00:49.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:49 smithi086 ceph-mon[25584]: Detected new or changed devices on smithi086 2024-03-04T16:00:49.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:49 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:49.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:49 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:49.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:49 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:49.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:49 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:49.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:49 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:49.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:49 smithi086 ceph-mon[25584]: Adjusting osd_memory_target on smithi086 to 4272M 2024-03-04T16:00:49.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:49 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:49.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:49 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:49.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:49 smithi086 ceph-mon[25584]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:00:49.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[24738]: osdmap e56: 8 total, 8 up, 8 in 2024-03-04T16:00:49.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[24738]: Detected new or changed devices on smithi086 2024-03-04T16:00:49.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:49.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:49.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:49.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:49.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:49.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[24738]: Adjusting osd_memory_target on smithi086 to 4272M 2024-03-04T16:00:49.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:49.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:49.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[24738]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:00:49.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[29452]: osdmap e56: 8 total, 8 up, 8 in 2024-03-04T16:00:49.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[29452]: Detected new or changed devices on smithi086 2024-03-04T16:00:49.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:49.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:49.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:49.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:49.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:49.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[29452]: Adjusting osd_memory_target on smithi086 to 4272M 2024-03-04T16:00:49.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:49.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:49.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:49 smithi083 ceph-mon[29452]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:00:50.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:50 smithi086 ceph-mon[25584]: osdmap e57: 8 total, 8 up, 8 in 2024-03-04T16:00:50.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:50 smithi086 ceph-mon[25584]: osdmap e58: 8 total, 8 up, 8 in 2024-03-04T16:00:50.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:50 smithi083 ceph-mon[24738]: osdmap e57: 8 total, 8 up, 8 in 2024-03-04T16:00:50.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:50 smithi083 ceph-mon[24738]: osdmap e58: 8 total, 8 up, 8 in 2024-03-04T16:00:50.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:50 smithi083 ceph-mon[29452]: osdmap e57: 8 total, 8 up, 8 in 2024-03-04T16:00:50.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:50 smithi083 ceph-mon[29452]: osdmap e58: 8 total, 8 up, 8 in 2024-03-04T16:00:50.624 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-04T16:00:50.624 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":58,"fsid":"98ffad3c-da3f-11ee-95c6-87774f69a715","created":"2024-03-04T15:56:00.302465+0000","modified":"2024-03-04T16:00:50.079277+0000","last_up_change":"2024-03-04T16:00:47.205316+0000","last_in_change":"2024-03-04T16:00:30.390767+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T15:58:53.056819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"87dbf1ea-c0a9-4218-8659-1120adebf7e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":3013656475},{"type":"v1","addr":"172.21.15.83:6803","nonce":3013656475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":3013656475},{"type":"v1","addr":"172.21.15.83:6805","nonce":3013656475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3013656475},{"type":"v1","addr":"172.21.15.83:6809","nonce":3013656475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":3013656475},{"type":"v1","addr":"172.21.15.83:6807","nonce":3013656475}]},"public_addr":"172.21.15.83:6803/3013656475","cluster_addr":"172.21.15.83:6805/3013656475","heartbeat_back_addr":"172.21.15.83:6809/3013656475","heartbeat_front_addr":"172.21.15.83:6807/3013656475","state":["exists","up"]},{"osd":1,"uuid":"50c23dd2-2738-4b9b-8155-720d51c5a16c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2964268155},{"type":"v1","addr":"172.21.15.83:6811","nonce":2964268155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2964268155},{"type":"v1","addr":"172.21.15.83:6813","nonce":2964268155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2964268155},{"type":"v1","addr":"172.21.15.83:6817","nonce":2964268155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2964268155},{"type":"v1","addr":"172.21.15.83:6815","nonce":2964268155}]},"public_addr":"172.21.15.83:6811/2964268155","cluster_addr":"172.21.15.83:6813/2964268155","heartbeat_back_addr":"172.21.15.83:6817/2964268155","heartbeat_front_addr":"172.21.15.83:6815/2964268155","state":["exists","up"]},{"osd":2,"uuid":"6bf2ebd9-d898-49c4-befd-df999267dab5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1243652505},{"type":"v1","addr":"172.21.15.83:6819","nonce":1243652505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1243652505},{"type":"v1","addr":"172.21.15.83:6821","nonce":1243652505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":1243652505},{"type":"v1","addr":"172.21.15.83:6825","nonce":1243652505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1243652505},{"type":"v1","addr":"172.21.15.83:6823","nonce":1243652505}]},"public_addr":"172.21.15.83:6819/1243652505","cluster_addr":"172.21.15.83:6821/1243652505","heartbeat_back_addr":"172.21.15.83:6825/1243652505","heartbeat_front_addr":"172.21.15.83:6823/1243652505","state":["exists","up"]},{"osd":3,"uuid":"24508e11-465f-43e0-8aad-4ce41214b4b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1643785747},{"type":"v1","addr":"172.21.15.83:6827","nonce":1643785747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":1643785747},{"type":"v1","addr":"172.21.15.83:6829","nonce":1643785747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1643785747},{"type":"v1","addr":"172.21.15.83:6833","nonce":1643785747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":1643785747},{"type":"v1","addr":"172.21.15.83:6831","nonce":1643785747}]},"public_addr":"172.21.15.83:6827/1643785747","cluster_addr":"172.21.15.83:6829/1643785747","heartbeat_back_addr":"172.21.15.83:6833/1643785747","heartbeat_front_addr":"172.21.15.83:6831/1643785747","state":["exists","up"]},{"osd":4,"uuid":"79c9ab9a-0df7-485a-bba9-54470969695e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":745738669},{"type":"v1","addr":"172.21.15.86:6801","nonce":745738669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":745738669},{"type":"v1","addr":"172.21.15.86:6803","nonce":745738669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":745738669},{"type":"v1","addr":"172.21.15.86:6807","nonce":745738669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":745738669},{"type":"v1","addr":"172.21.15.86:6805","nonce":745738669}]},"public_addr":"172.21.15.86:6801/745738669","cluster_addr":"172.21.15.86:6803/745738669","heartbeat_back_addr":"172.21.15.86:6807/745738669","heartbeat_front_addr":"172.21.15.86:6805/745738669","state":["exists","up"]},{"osd":5,"uuid":"0e1dc2f0-4696-49d9-8691-109139cc3558","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":3306461240},{"type":"v1","addr":"172.21.15.86:6809","nonce":3306461240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":3306461240},{"type":"v1","addr":"172.21.15.86:6811","nonce":3306461240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":3306461240},{"type":"v1","addr":"172.21.15.86:6815","nonce":3306461240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":3306461240},{"type":"v1","addr":"172.21.15.86:6813","nonce":3306461240}]},"public_addr":"172.21.15.86:6809/3306461240","cluster_addr":"172.21.15.86:6811/3306461240","heartbeat_back_addr":"172.21.15.86:6815/3306461240","heartbeat_front_addr":"172.21.15.86:6813/3306461240","state":["exists","up"]},{"osd":6,"uuid":"bd5216b0-8187-40f2-a10f-724f8f8c95a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":3999097134},{"type":"v1","addr":"172.21.15.86:6817","nonce":3999097134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3999097134},{"type":"v1","addr":"172.21.15.86:6819","nonce":3999097134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3999097134},{"type":"v1","addr":"172.21.15.86:6823","nonce":3999097134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3999097134},{"type":"v1","addr":"172.21.15.86:6821","nonce":3999097134}]},"public_addr":"172.21.15.86:6817/3999097134","cluster_addr":"172.21.15.86:6819/3999097134","heartbeat_back_addr":"172.21.15.86:6823/3999097134","heartbeat_front_addr":"172.21.15.86:6821/3999097134","state":["exists","up"]},{"osd":7,"uuid":"11856cf1-448a-4a87-822a-4df57524fa47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":702851952},{"type":"v1","addr":"172.21.15.86:6825","nonce":702851952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":702851952},{"type":"v1","addr":"172.21.15.86:6827","nonce":702851952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":702851952},{"type":"v1","addr":"172.21.15.86:6831","nonce":702851952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":702851952},{"type":"v1","addr":"172.21.15.86:6829","nonce":702851952}]},"public_addr":"172.21.15.86:6825/702851952","cluster_addr":"172.21.15.86:6827/702851952","heartbeat_back_addr":"172.21.15.86:6831/702851952","heartbeat_front_addr":"172.21.15.86:6829/702851952","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:03.296875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:26.160457+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:49.263749+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:13.395869+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:35.202478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:59.251797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:00:22.154238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:00:45.756613+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:6800/645025785":"2024-03-05T15:56:44.688018+0000","172.21.15.83:0/1017961836":"2024-03-05T15:56:44.688018+0000","172.21.15.83:0/1054338912":"2024-03-05T15:56:44.688018+0000","172.21.15.83:6800/2606489355":"2024-03-05T15:56:22.880818+0000","172.21.15.83:6801/2606489355":"2024-03-05T15:56:22.880818+0000","172.21.15.83:6801/645025785":"2024-03-05T15:56:44.688018+0000","172.21.15.83:0/2983002978":"2024-03-05T15:56:22.880818+0000","172.21.15.83:0/489466195":"2024-03-05T15:56:44.688018+0000","172.21.15.83:0/1474787565":"2024-03-05T15:56:22.880818+0000","172.21.15.83:0/106501873":"2024-03-05T15:56:22.880818+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:00:51.363 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:51 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/4279924862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:00:51.363 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:51 smithi083 ceph-mon[24738]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:00:51.363 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:51 smithi083 ceph-mon[24738]: osdmap e59: 8 total, 8 up, 8 in 2024-03-04T16:00:51.363 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:51 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/4279924862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:00:51.363 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:51 smithi083 ceph-mon[29452]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:00:51.364 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:51 smithi083 ceph-mon[29452]: osdmap e59: 8 total, 8 up, 8 in 2024-03-04T16:00:51.387 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T15:58:53.056819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-04T16:00:51.387 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-04T16:00:51.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:51 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/4279924862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:00:51.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:51 smithi086 ceph-mon[25584]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:00:51.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:51 smithi086 ceph-mon[25584]: osdmap e59: 8 total, 8 up, 8 in 2024-03-04T16:00:51.601 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:00:53.443 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-04T16:00:54.053 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:53 smithi083 ceph-mon[24738]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:00:54.053 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:53 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3883340681' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T16:00:54.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:53 smithi083 ceph-mon[29452]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:00:54.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:53 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3883340681' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T16:00:54.076 INFO:tasks.cephadm:Adding prometheus.a on smithi086 2024-03-04T16:00:54.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph orch apply prometheus '1;smithi086=a' 2024-03-04T16:00:54.106 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:53 smithi086 ceph-mon[25584]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:00:54.106 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:53 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3883340681' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T16:00:55.851 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled prometheus update... 2024-03-04T16:00:55.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:55 smithi086 ceph-mon[25584]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-04T16:00:56.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:55 smithi083 ceph-mon[24738]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-04T16:00:56.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:55 smithi083 ceph-mon[29452]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-04T16:00:56.462 DEBUG:teuthology.orchestra.run.smithi086:prometheus.a> sudo journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@prometheus.a.service 2024-03-04T16:00:56.504 INFO:tasks.cephadm:Adding node-exporter.a on smithi083 2024-03-04T16:00:56.504 INFO:tasks.cephadm:Adding node-exporter.b on smithi086 2024-03-04T16:00:56.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph orch apply node-exporter '2;smithi083=a;smithi086=b' 2024-03-04T16:00:56.985 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:00:56 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ignoring --setuser ceph since I am not root 2024-03-04T16:00:56.985 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:00:56 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ignoring --setgroup ceph since I am not root 2024-03-04T16:00:56.985 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:00:56 smithi086 ceph-mgr[26681]: -- 172.21.15.86:0/2035444222 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5602ab4ae340 con 0x5602ab486800 2024-03-04T16:00:56.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:56 smithi086 ceph-mon[25584]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:56.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:56 smithi086 ceph-mon[25584]: Saving service prometheus spec with placement smithi086=a;count:1 2024-03-04T16:00:56.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:56 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:56.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:56 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:56.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:56 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:56.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:56 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:56.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:56 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:56.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:56 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-04T16:00:57.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:00:56 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ignoring --setuser ceph since I am not root 2024-03-04T16:00:57.239 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:00:56 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ignoring --setgroup ceph since I am not root 2024-03-04T16:00:57.239 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:00:56 smithi083 ceph-mgr[24974]: -- 172.21.15.83:0/1196896648 <== mon.1 v2:172.21.15.83:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c55bd74340 con 0x55c55bd4cc00 2024-03-04T16:00:57.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:56 smithi083 ceph-mon[24738]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:57.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:56 smithi083 ceph-mon[24738]: Saving service prometheus spec with placement smithi086=a;count:1 2024-03-04T16:00:57.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:56 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:57.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:56 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:57.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:56 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:57.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:56 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:57.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:56 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:57.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:56 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-04T16:00:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:56 smithi083 ceph-mon[29452]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:56 smithi083 ceph-mon[29452]: Saving service prometheus spec with placement smithi086=a;count:1 2024-03-04T16:00:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:56 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:56 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:56 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:56 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:56 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' 2024-03-04T16:00:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:56 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-04T16:00:57.668 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:00:57 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:00:57.408+0000 7f65fe923000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T16:00:57.668 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:00:57 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:00:57.498+0000 7f65fe923000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T16:00:57.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:00:57 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:00:57.388+0000 7fa67559c000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T16:00:57.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:00:57 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:00:57.478+0000 7fa67559c000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T16:00:57.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:57 smithi086 ceph-mon[25584]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-04T16:00:57.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:57 smithi086 ceph-mon[25584]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-04T16:00:57.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:00:57 smithi086 ceph-mon[25584]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-04T16:00:58.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:57 smithi083 ceph-mon[24738]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-04T16:00:58.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:57 smithi083 ceph-mon[24738]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-04T16:00:58.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:00:57 smithi083 ceph-mon[24738]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-04T16:00:58.151 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:00:58 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:00:58.062+0000 7fa67559c000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T16:00:58.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:57 smithi083 ceph-mon[29452]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-04T16:00:58.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:57 smithi083 ceph-mon[29452]: from='mgr.14150 172.21.15.83:0/1875519064' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-04T16:00:58.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:00:57 smithi083 ceph-mon[29452]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-04T16:00:58.278 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:00:58 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:00:58.104+0000 7f65fe923000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T16:00:58.278 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:00:58 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:00:58.202+0000 7f65fe923000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T16:00:58.463 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:00:58 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:00:58.149+0000 7fa67559c000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T16:00:58.463 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:00:58 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:00:58.233+0000 7fa67559c000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T16:00:58.463 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:00:58 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:00:58.382+0000 7fa67559c000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T16:00:58.464 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:00:58 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:00:58.462+0000 7fa67559c000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T16:00:58.631 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:00:58 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:00:58.285+0000 7f65fe923000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T16:00:58.631 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:00:58 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:00:58.439+0000 7f65fe923000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T16:00:58.631 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:00:58 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:00:58.520+0000 7f65fe923000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T16:00:58.631 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:00:58 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:00:58.629+0000 7f65fe923000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T16:00:58.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:00:58 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:00:58.569+0000 7fa67559c000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T16:00:58.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:00:58 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:00:58.653+0000 7fa67559c000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T16:00:58.955 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:00:58 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:00:58.712+0000 7f65fe923000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T16:00:58.955 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:00:58 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:00:58.871+0000 7f65fe923000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T16:00:59.166 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:00:58 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:00:58.815+0000 7fa67559c000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T16:00:59.166 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:00:58 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:00:58.900+0000 7fa67559c000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T16:00:59.234 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:00:58 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:00:58.954+0000 7f65fe923000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T16:01:00.069 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:00:59 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:00:59.940+0000 7fa67559c000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T16:01:00.234 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:00:59 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:00:59.963+0000 7f65fe923000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T16:01:00.234 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:00 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:01:00.093+0000 7f65fe923000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T16:01:00.347 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:00 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:01:00.068+0000 7fa67559c000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T16:01:00.347 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:00 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:01:00.259+0000 7fa67559c000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T16:01:00.347 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:00 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:01:00.345+0000 7fa67559c000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T16:01:00.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:00 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:01:00.281+0000 7f65fe923000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T16:01:00.735 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:00 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:01:00.368+0000 7f65fe923000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T16:01:00.735 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:00 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:01:00.503+0000 7f65fe923000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:01:00.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:00 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:01:00.481+0000 7fa67559c000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:01:01.105 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:00 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:01:00.797+0000 7fa67559c000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T16:01:01.105 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:01 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:01:01.018+0000 7fa67559c000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:01:01.105 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:01 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:01:01.104+0000 7fa67559c000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T16:01:01.115 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:00 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:01:00.816+0000 7f65fe923000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T16:01:01.115 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:01 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:01:01.032+0000 7f65fe923000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:01:01.116 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:01 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:01:01.115+0000 7f65fe923000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T16:01:01.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:01 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:01:01.282+0000 7f65fe923000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T16:01:01.488 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:01 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:01:01.274+0000 7fa67559c000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T16:01:02.480 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:02 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:01:02.191+0000 7f65fe923000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T16:01:02.480 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:02 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:01:02.479+0000 7f65fe923000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T16:01:02.486 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:01:02.190+0000 7fa67559c000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T16:01:02.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:02 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:01:02.572+0000 7f65fe923000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T16:01:02.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:02 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:01:02.723+0000 7f65fe923000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T16:01:02.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:01:02.485+0000 7fa67559c000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T16:01:02.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:01:02.582+0000 7fa67559c000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T16:01:02.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:01:02.733+0000 7fa67559c000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T16:01:03.127 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: [04/Mar/2024:16:01:02] ENGINE Bus STARTING 2024-03-04T16:01:03.128 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-mon[29452]: Standby manager daemon x restarted 2024-03-04T16:01:03.128 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-mon[29452]: Standby manager daemon x started 2024-03-04T16:01:03.128 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/3427070862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T16:01:03.128 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/3427070862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:01:03.128 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/3427070862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T16:01:03.128 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/3427070862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:01:03.128 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-mon[29452]: Active manager daemon y restarted 2024-03-04T16:01:03.128 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-mon[29452]: Activating manager daemon y 2024-03-04T16:01:03.128 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-mon[29452]: osdmap e60: 8 total, 8 up, 8 in 2024-03-04T16:01:03.129 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-mon[24738]: Standby manager daemon x restarted 2024-03-04T16:01:03.129 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-mon[24738]: Standby manager daemon x started 2024-03-04T16:01:03.129 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/3427070862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T16:01:03.129 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/3427070862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:01:03.129 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/3427070862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T16:01:03.129 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/3427070862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:01:03.129 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-mon[24738]: Active manager daemon y restarted 2024-03-04T16:01:03.129 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-mon[24738]: Activating manager daemon y 2024-03-04T16:01:03.129 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:02 smithi083 ceph-mon[24738]: osdmap e60: 8 total, 8 up, 8 in 2024-03-04T16:01:03.171 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:02 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: [04/Mar/2024:16:01:02] ENGINE Bus STARTING 2024-03-04T16:01:03.171 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:02 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: CherryPy Checker: 2024-03-04T16:01:03.171 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:02 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: The Application mounted at '' has an empty config. 2024-03-04T16:01:03.172 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:02 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: 2024-03-04T16:01:03.172 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:02 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: [04/Mar/2024:16:01:02] ENGINE Serving on http://:::9283 2024-03-04T16:01:03.172 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:02 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: [04/Mar/2024:16:01:02] ENGINE Bus STARTED 2024-03-04T16:01:03.172 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:02 smithi086 ceph-mon[25584]: Standby manager daemon x restarted 2024-03-04T16:01:03.172 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:02 smithi086 ceph-mon[25584]: Standby manager daemon x started 2024-03-04T16:01:03.172 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:02 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/3427070862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T16:01:03.172 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:02 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/3427070862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:01:03.172 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:02 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/3427070862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T16:01:03.172 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:02 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/3427070862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:01:03.172 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:02 smithi086 ceph-mon[25584]: Active manager daemon y restarted 2024-03-04T16:01:03.173 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:02 smithi086 ceph-mon[25584]: Activating manager daemon y 2024-03-04T16:01:03.173 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:02 smithi086 ceph-mon[25584]: osdmap e60: 8 total, 8 up, 8 in 2024-03-04T16:01:03.394 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: CherryPy Checker: 2024-03-04T16:01:03.394 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: The Application mounted at '' has an empty config. 2024-03-04T16:01:03.394 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:01:03.394 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: [04/Mar/2024:16:01:03] ENGINE Serving on http://:::9283 2024-03-04T16:01:03.394 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: [04/Mar/2024:16:01:03] ENGINE Bus STARTED 2024-03-04T16:01:03.739 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: [04/Mar/2024:16:01:03] ENGINE Bus STARTING 2024-03-04T16:01:03.739 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: [04/Mar/2024:16:01:03] ENGINE Serving on https://172.21.15.83:7150 2024-03-04T16:01:03.739 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: [04/Mar/2024:16:01:03] ENGINE Bus STARTED 2024-03-04T16:01:03.818 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled node-exporter update... 2024-03-04T16:01:04.159 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: mgrmap e17: y(active, starting, since 0.0451554s), standbys: x 2024-03-04T16:01:04.159 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:01:04.159 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:01:04.159 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:01:04.159 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T16:01:04.159 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T16:01:04.159 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:01:04.159 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:01:04.160 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:01:04.160 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:01:04.160 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:04.160 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:04.160 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:01:04.160 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:01:04.160 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:01:04.160 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:01:04.160 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:01:04.160 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: Manager daemon y is now available 2024-03-04T16:01:04.160 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:04.160 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:04.160 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:04.161 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:04.161 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:04.161 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:01:04.161 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:01:04.161 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:01:04.161 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:01:04.161 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:03 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:04.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: mgrmap e17: y(active, starting, since 0.0451554s), standbys: x 2024-03-04T16:01:04.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:01:04.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:01:04.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:01:04.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T16:01:04.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T16:01:04.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:01:04.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:01:04.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:01:04.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:01:04.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:04.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:04.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:01:04.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:01:04.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:01:04.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:01:04.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:01:04.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: Manager daemon y is now available 2024-03-04T16:01:04.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:04.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:04.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:04.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:04.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:04.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:01:04.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:01:04.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:01:04.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:01:04.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:04.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: mgrmap e17: y(active, starting, since 0.0451554s), standbys: x 2024-03-04T16:01:04.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:01:04.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:01:04.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:01:04.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T16:01:04.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T16:01:04.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:01:04.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:01:04.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:01:04.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:01:04.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:04.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:04.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:01:04.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:01:04.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:01:04.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:01:04.243 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:01:04.243 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: Manager daemon y is now available 2024-03-04T16:01:04.243 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:04.243 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:04.243 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:04.243 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:04.243 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:04.243 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:01:04.243 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:01:04.243 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:01:04.243 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:01:04.243 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:03 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:04.483 DEBUG:teuthology.orchestra.run.smithi083:node-exporter.a> sudo journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@node-exporter.a.service 2024-03-04T16:01:04.485 DEBUG:teuthology.orchestra.run.smithi086:node-exporter.b> sudo journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@node-exporter.b.service 2024-03-04T16:01:04.488 INFO:tasks.cephadm:Adding alertmanager.a on smithi083 2024-03-04T16:01:04.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph orch apply alertmanager '1;smithi083=a' 2024-03-04T16:01:05.073 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:04 smithi083 ceph-mon[29452]: [04/Mar/2024:16:01:03] ENGINE Bus STARTING 2024-03-04T16:01:05.073 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:04 smithi083 ceph-mon[29452]: [04/Mar/2024:16:01:03] ENGINE Serving on https://172.21.15.83:7150 2024-03-04T16:01:05.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:04 smithi083 ceph-mon[29452]: [04/Mar/2024:16:01:03] ENGINE Bus STARTED 2024-03-04T16:01:05.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:04 smithi083 ceph-mon[29452]: mgrmap e18: y(active, since 1.06148s), standbys: x 2024-03-04T16:01:05.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:04 smithi083 ceph-mon[29452]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi083=a;smithi086=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:05.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:04 smithi083 ceph-mon[29452]: Saving service node-exporter spec with placement smithi083=a;smithi086=b;count:2 2024-03-04T16:01:05.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:04 smithi083 ceph-mon[29452]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:05.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:04 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:05.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:04 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:05.074 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:04 smithi083 ceph-mon[24738]: [04/Mar/2024:16:01:03] ENGINE Bus STARTING 2024-03-04T16:01:05.074 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:04 smithi083 ceph-mon[24738]: [04/Mar/2024:16:01:03] ENGINE Serving on https://172.21.15.83:7150 2024-03-04T16:01:05.074 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:04 smithi083 ceph-mon[24738]: [04/Mar/2024:16:01:03] ENGINE Bus STARTED 2024-03-04T16:01:05.074 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:04 smithi083 ceph-mon[24738]: mgrmap e18: y(active, since 1.06148s), standbys: x 2024-03-04T16:01:05.075 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:04 smithi083 ceph-mon[24738]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi083=a;smithi086=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:05.075 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:04 smithi083 ceph-mon[24738]: Saving service node-exporter spec with placement smithi083=a;smithi086=b;count:2 2024-03-04T16:01:05.075 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:04 smithi083 ceph-mon[24738]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:05.075 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:04 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:05.075 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:04 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:05.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:04 smithi086 ceph-mon[25584]: [04/Mar/2024:16:01:03] ENGINE Bus STARTING 2024-03-04T16:01:05.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:04 smithi086 ceph-mon[25584]: [04/Mar/2024:16:01:03] ENGINE Serving on https://172.21.15.83:7150 2024-03-04T16:01:05.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:04 smithi086 ceph-mon[25584]: [04/Mar/2024:16:01:03] ENGINE Bus STARTED 2024-03-04T16:01:05.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:04 smithi086 ceph-mon[25584]: mgrmap e18: y(active, since 1.06148s), standbys: x 2024-03-04T16:01:05.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:04 smithi086 ceph-mon[25584]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi083=a;smithi086=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:05.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:04 smithi086 ceph-mon[25584]: Saving service node-exporter spec with placement smithi083=a;smithi086=b;count:2 2024-03-04T16:01:05.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:04 smithi086 ceph-mon[25584]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:05.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:04 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:05.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:04 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:06.072 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:05 smithi086 ceph-mon[25584]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:06.072 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:05 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:06.072 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:05 smithi086 ceph-mon[25584]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-04T16:01:06.073 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:05 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:06.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:05 smithi083 ceph-mon[24738]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:06.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:05 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:06.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:05 smithi083 ceph-mon[24738]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-04T16:01:06.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:05 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:06.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:05 smithi083 ceph-mon[29452]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:06.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:05 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:06.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:05 smithi083 ceph-mon[29452]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-04T16:01:06.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:05 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:06.664 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled alertmanager update... 2024-03-04T16:01:06.877 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:06 smithi086 ceph-mon[25584]: Updating smithi083:/etc/ceph/ceph.conf 2024-03-04T16:01:06.877 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:06 smithi086 ceph-mon[25584]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:01:06.877 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:06 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:06.877 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:06 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:07.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:06 smithi083 ceph-mon[24738]: Updating smithi083:/etc/ceph/ceph.conf 2024-03-04T16:01:07.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:06 smithi083 ceph-mon[24738]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:01:07.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:06 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:07.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:06 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:07.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:06 smithi083 ceph-mon[29452]: Updating smithi083:/etc/ceph/ceph.conf 2024-03-04T16:01:07.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:06 smithi083 ceph-mon[29452]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:01:07.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:06 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:07.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:06 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:07.348 DEBUG:teuthology.orchestra.run.smithi083:alertmanager.a> sudo journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@alertmanager.a.service 2024-03-04T16:01:07.351 INFO:tasks.cephadm:Adding grafana.a on smithi086 2024-03-04T16:01:07.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph orch apply grafana '1;smithi086=a' 2024-03-04T16:01:07.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:07 smithi086 ceph-mon[25584]: from='client.34349 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi083=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:07.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:07 smithi086 ceph-mon[25584]: Saving service alertmanager spec with placement smithi083=a;count:1 2024-03-04T16:01:07.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:07 smithi086 ceph-mon[25584]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:07.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:07 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:07.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:07 smithi086 ceph-mon[25584]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-04T16:01:07.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:07 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:07.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:07 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:08.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:07 smithi083 ceph-mon[24738]: from='client.34349 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi083=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:08.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:07 smithi083 ceph-mon[24738]: Saving service alertmanager spec with placement smithi083=a;count:1 2024-03-04T16:01:08.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:07 smithi083 ceph-mon[24738]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:08.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:07 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:08.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:07 smithi083 ceph-mon[24738]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-04T16:01:08.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:07 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:08.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:07 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:08.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:07 smithi083 ceph-mon[29452]: from='client.34349 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi083=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:08.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:07 smithi083 ceph-mon[29452]: Saving service alertmanager spec with placement smithi083=a;count:1 2024-03-04T16:01:08.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:07 smithi083 ceph-mon[29452]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:08.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:07 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:08.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:07 smithi083 ceph-mon[29452]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-04T16:01:08.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:07 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:08.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:07 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:09.179 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:08 smithi086 ceph-mon[25584]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:01:09.180 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:08 smithi086 ceph-mon[25584]: Deploying daemon prometheus.a on smithi086 2024-03-04T16:01:09.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:08 smithi083 ceph-mon[29452]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:01:09.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:08 smithi083 ceph-mon[29452]: Deploying daemon prometheus.a on smithi086 2024-03-04T16:01:09.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:08 smithi083 ceph-mon[24738]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:01:09.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:08 smithi083 ceph-mon[24738]: Deploying daemon prometheus.a on smithi086 2024-03-04T16:01:09.353 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled grafana update... 2024-03-04T16:01:10.158 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:09 smithi083 ceph-mon[24738]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:10.158 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:09 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:10.158 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:09 smithi083 ceph-mon[29452]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:10.158 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:09 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:09 smithi086 ceph-mon[25584]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:09 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:10.523 DEBUG:teuthology.orchestra.run.smithi086:grafana.a> sudo journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@grafana.a.service 2024-03-04T16:01:10.525 INFO:tasks.cephadm:Setting up client nodes... 2024-03-04T16:01:10.528 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-04T16:01:11.203 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:10 smithi083 ceph-mon[29452]: from='client.34355 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:11.203 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:10 smithi083 ceph-mon[29452]: Saving service grafana spec with placement smithi086=a;count:1 2024-03-04T16:01:11.204 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:10 smithi083 ceph-mon[24738]: from='client.34355 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:11.204 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:10 smithi083 ceph-mon[24738]: Saving service grafana spec with placement smithi086=a;count:1 2024-03-04T16:01:11.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:10 smithi086 ceph-mon[25584]: from='client.34355 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:11.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:10 smithi086 ceph-mon[25584]: Saving service grafana spec with placement smithi086=a;count:1 2024-03-04T16:01:12.133 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:11 smithi083 ceph-mon[24738]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:12.134 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:11 smithi083 ceph-mon[29452]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:11 smithi086 ceph-mon[25584]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:12.647 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-03-04T16:01:12.647 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBI8OVlpelUJhAA1fag0JgxRVthkxZp3qJCZQ== 2024-03-04T16:01:12.980 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:12 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3275822719' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:01:12.981 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:12 smithi083 ceph-mon[24738]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:01:12.981 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:12 smithi083 ceph-mon[24738]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T16:01:12.981 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:12 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3275822719' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:01:12.981 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:12 smithi083 ceph-mon[29452]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:01:12.981 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:12 smithi083 ceph-mon[29452]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T16:01:13.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:12 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3275822719' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:01:13.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:12 smithi086 ceph-mon[25584]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:01:13.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:12 smithi086 ceph-mon[25584]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T16:01:13.288 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-04T16:01:13.288 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-04T16:01:13.288 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-04T16:01:13.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-04T16:01:14.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:13 smithi086 ceph-mon[25584]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:14.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:13 smithi083 ceph-mon[24738]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:14.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:13 smithi083 ceph-mon[29452]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:16.130 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:15 smithi086 ceph-mon[25584]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:16.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:15 smithi083 ceph-mon[29452]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:16.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:15 smithi083 ceph-mon[24738]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:18.201 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:17 smithi086 ceph-mon[25584]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:18.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:17 smithi083 ceph-mon[24738]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:18.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:17 smithi083 ceph-mon[29452]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:18.772 INFO:teuthology.orchestra.run.smithi086.stdout:[client.1] 2024-03-04T16:01:18.772 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQBO8OVlnMbDLRAAlDz4IwN0k/GzUYQIIbZS9g== 2024-03-04T16:01:19.029 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:18 smithi086 ceph-mon[25584]: from='client.? 172.21.15.86:0/1482422715' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:01:19.030 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:18 smithi086 ceph-mon[25584]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:01:19.030 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:18 smithi086 ceph-mon[25584]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T16:01:19.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:18 smithi083 ceph-mon[24738]: from='client.? 172.21.15.86:0/1482422715' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:01:19.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:18 smithi083 ceph-mon[24738]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:01:19.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:18 smithi083 ceph-mon[24738]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T16:01:19.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:18 smithi083 ceph-mon[29452]: from='client.? 172.21.15.86:0/1482422715' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:01:19.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:18 smithi083 ceph-mon[29452]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:01:19.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:18 smithi083 ceph-mon[29452]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T16:01:19.947 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-04T16:01:19.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-04T16:01:19.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-04T16:01:19.995 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-04T16:01:19.996 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-04T16:01:19.996 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph mgr dump --format=json 2024-03-04T16:01:20.160 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:19 smithi086 ceph-mon[25584]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:20.219 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:01:20.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:19 smithi083 ceph-mon[24738]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:20.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:19 smithi083 ceph-mon[29452]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:20.485 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:20 smithi086 podman[40601]: 2024-03-04 16:01:20.063212634 +0000 UTC m=+0.018564347 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-04T16:01:21.314 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-mon[25584]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:21.315 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 podman[40601]: 2024-03-04 16:01:21.058306915 +0000 UTC m=+1.013658627 container create 54f5ce4438a69e5889c06310872a3d0f292592714800309e39b3f69a01b7c614 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:01:21.315 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 podman[40601]: 2024-03-04 16:01:21.260825065 +0000 UTC m=+1.216176776 container init 54f5ce4438a69e5889c06310872a3d0f292592714800309e39b3f69a01b7c614 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:01:21.315 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 podman[40601]: 2024-03-04 16:01:21.263916294 +0000 UTC m=+1.219268014 container start 54f5ce4438a69e5889c06310872a3d0f292592714800309e39b3f69a01b7c614 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:01:21.315 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:01:21.290Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-04T16:01:21.315 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:01:21.290Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-04T16:01:21.315 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:01:21.290Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-04T16:01:21.315 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:01:21.290Z caller=main.go:518 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi086 (none))" 2024-03-04T16:01:21.315 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:01:21.290Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-04T16:01:21.315 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:01:21.290Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-04T16:01:21.315 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:01:21.292Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-04T16:01:21.315 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:01:21.292Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-04T16:01:21.315 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:01:21.293Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-04T16:01:21.316 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:01:21.296Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-04T16:01:21.316 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:01:21.296Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.038µs 2024-03-04T16:01:21.316 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:01:21.296Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-04T16:01:21.316 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:01:21.297Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-04T16:01:21.316 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:01:21.297Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=21.968µs wal_replay_duration=377.329µs total_replay_duration=411.623µs 2024-03-04T16:01:21.316 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:01:21.298Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-04T16:01:21.316 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:01:21.298Z caller=main.go:947 level=info msg="TSDB started" 2024-03-04T16:01:21.316 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:01:21.298Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-04T16:01:21.316 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:01:21.313Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.326719ms db_storage=786ns remote_storage=1.359µs web_handler=260ns query_engine=619ns scrape=408.962µs scrape_sd=18.51µs notify=902ns notify_sd=1.403µs rules=14.731735ms 2024-03-04T16:01:21.316 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:01:21.313Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-04T16:01:21.377 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:21 smithi083 ceph-mon[24738]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:21.377 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:21 smithi083 ceph-mon[29452]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:21.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 bash[40601]: 54f5ce4438a69e5889c06310872a3d0f292592714800309e39b3f69a01b7c614 2024-03-04T16:01:21.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:01:21 smithi086 systemd[1]: Started Ceph prometheus.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:01:22.234 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-04T16:01:22.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:22 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:22.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:22 smithi083 ceph-mon[24738]: Deploying daemon node-exporter.a on smithi083 2024-03-04T16:01:22.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:22 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3689476020' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T16:01:22.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:22 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:22.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:22 smithi083 ceph-mon[29452]: Deploying daemon node-exporter.a on smithi083 2024-03-04T16:01:22.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:22 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3689476020' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T16:01:22.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:22 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:22.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:22 smithi086 ceph-mon[25584]: Deploying daemon node-exporter.a on smithi083 2024-03-04T16:01:22.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:22 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3689476020' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T16:01:22.861 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":19,"active_gid":24337,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1219817995},{"type":"v1","addr":"172.21.15.83:6801","nonce":1219817995}]},"active_addr":"172.21.15.83:6801/1219817995","active_change":"2024-03-04T16:01:02.747639+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34310,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.83:8443/","prometheus":"http://172.21.15.83:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":60,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":1809332442}]},{"addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":2796988943}]},{"addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":2940780554}]},{"addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":853568958}]}]}} 2024-03-04T16:01:22.865 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-04T16:01:22.865 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-04T16:01:22.865 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T16:01:23.100 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:01:23.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:23 smithi083 ceph-mon[24738]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:23.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:23 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:23.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:23 smithi083 ceph-mon[29452]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:23.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:23 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:24.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:23 smithi086 ceph-mon[25584]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:24.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:23 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:25.894 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-04T16:01:25.895 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":60,"fsid":"98ffad3c-da3f-11ee-95c6-87774f69a715","created":"2024-03-04T15:56:00.302465+0000","modified":"2024-03-04T16:01:02.747283+0000","last_up_change":"2024-03-04T16:00:47.205316+0000","last_in_change":"2024-03-04T16:00:30.390767+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T15:58:53.056819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"87dbf1ea-c0a9-4218-8659-1120adebf7e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":3013656475},{"type":"v1","addr":"172.21.15.83:6803","nonce":3013656475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":3013656475},{"type":"v1","addr":"172.21.15.83:6805","nonce":3013656475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3013656475},{"type":"v1","addr":"172.21.15.83:6809","nonce":3013656475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":3013656475},{"type":"v1","addr":"172.21.15.83:6807","nonce":3013656475}]},"public_addr":"172.21.15.83:6803/3013656475","cluster_addr":"172.21.15.83:6805/3013656475","heartbeat_back_addr":"172.21.15.83:6809/3013656475","heartbeat_front_addr":"172.21.15.83:6807/3013656475","state":["exists","up"]},{"osd":1,"uuid":"50c23dd2-2738-4b9b-8155-720d51c5a16c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2964268155},{"type":"v1","addr":"172.21.15.83:6811","nonce":2964268155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2964268155},{"type":"v1","addr":"172.21.15.83:6813","nonce":2964268155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2964268155},{"type":"v1","addr":"172.21.15.83:6817","nonce":2964268155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2964268155},{"type":"v1","addr":"172.21.15.83:6815","nonce":2964268155}]},"public_addr":"172.21.15.83:6811/2964268155","cluster_addr":"172.21.15.83:6813/2964268155","heartbeat_back_addr":"172.21.15.83:6817/2964268155","heartbeat_front_addr":"172.21.15.83:6815/2964268155","state":["exists","up"]},{"osd":2,"uuid":"6bf2ebd9-d898-49c4-befd-df999267dab5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1243652505},{"type":"v1","addr":"172.21.15.83:6819","nonce":1243652505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1243652505},{"type":"v1","addr":"172.21.15.83:6821","nonce":1243652505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":1243652505},{"type":"v1","addr":"172.21.15.83:6825","nonce":1243652505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1243652505},{"type":"v1","addr":"172.21.15.83:6823","nonce":1243652505}]},"public_addr":"172.21.15.83:6819/1243652505","cluster_addr":"172.21.15.83:6821/1243652505","heartbeat_back_addr":"172.21.15.83:6825/1243652505","heartbeat_front_addr":"172.21.15.83:6823/1243652505","state":["exists","up"]},{"osd":3,"uuid":"24508e11-465f-43e0-8aad-4ce41214b4b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1643785747},{"type":"v1","addr":"172.21.15.83:6827","nonce":1643785747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":1643785747},{"type":"v1","addr":"172.21.15.83:6829","nonce":1643785747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1643785747},{"type":"v1","addr":"172.21.15.83:6833","nonce":1643785747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":1643785747},{"type":"v1","addr":"172.21.15.83:6831","nonce":1643785747}]},"public_addr":"172.21.15.83:6827/1643785747","cluster_addr":"172.21.15.83:6829/1643785747","heartbeat_back_addr":"172.21.15.83:6833/1643785747","heartbeat_front_addr":"172.21.15.83:6831/1643785747","state":["exists","up"]},{"osd":4,"uuid":"79c9ab9a-0df7-485a-bba9-54470969695e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":745738669},{"type":"v1","addr":"172.21.15.86:6801","nonce":745738669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":745738669},{"type":"v1","addr":"172.21.15.86:6803","nonce":745738669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":745738669},{"type":"v1","addr":"172.21.15.86:6807","nonce":745738669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":745738669},{"type":"v1","addr":"172.21.15.86:6805","nonce":745738669}]},"public_addr":"172.21.15.86:6801/745738669","cluster_addr":"172.21.15.86:6803/745738669","heartbeat_back_addr":"172.21.15.86:6807/745738669","heartbeat_front_addr":"172.21.15.86:6805/745738669","state":["exists","up"]},{"osd":5,"uuid":"0e1dc2f0-4696-49d9-8691-109139cc3558","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":3306461240},{"type":"v1","addr":"172.21.15.86:6809","nonce":3306461240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":3306461240},{"type":"v1","addr":"172.21.15.86:6811","nonce":3306461240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":3306461240},{"type":"v1","addr":"172.21.15.86:6815","nonce":3306461240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":3306461240},{"type":"v1","addr":"172.21.15.86:6813","nonce":3306461240}]},"public_addr":"172.21.15.86:6809/3306461240","cluster_addr":"172.21.15.86:6811/3306461240","heartbeat_back_addr":"172.21.15.86:6815/3306461240","heartbeat_front_addr":"172.21.15.86:6813/3306461240","state":["exists","up"]},{"osd":6,"uuid":"bd5216b0-8187-40f2-a10f-724f8f8c95a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":3999097134},{"type":"v1","addr":"172.21.15.86:6817","nonce":3999097134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3999097134},{"type":"v1","addr":"172.21.15.86:6819","nonce":3999097134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3999097134},{"type":"v1","addr":"172.21.15.86:6823","nonce":3999097134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3999097134},{"type":"v1","addr":"172.21.15.86:6821","nonce":3999097134}]},"public_addr":"172.21.15.86:6817/3999097134","cluster_addr":"172.21.15.86:6819/3999097134","heartbeat_back_addr":"172.21.15.86:6823/3999097134","heartbeat_front_addr":"172.21.15.86:6821/3999097134","state":["exists","up"]},{"osd":7,"uuid":"11856cf1-448a-4a87-822a-4df57524fa47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":702851952},{"type":"v1","addr":"172.21.15.86:6825","nonce":702851952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":702851952},{"type":"v1","addr":"172.21.15.86:6827","nonce":702851952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":702851952},{"type":"v1","addr":"172.21.15.86:6831","nonce":702851952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":702851952},{"type":"v1","addr":"172.21.15.86:6829","nonce":702851952}]},"public_addr":"172.21.15.86:6825/702851952","cluster_addr":"172.21.15.86:6827/702851952","heartbeat_back_addr":"172.21.15.86:6831/702851952","heartbeat_front_addr":"172.21.15.86:6829/702851952","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:03.296875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:26.160457+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:49.263749+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:13.395869+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:35.202478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:59.251797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:00:22.154238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:00:45.756613+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:0/1947529438":"2024-03-05T16:01:02.747243+0000","172.21.15.83:0/3984588694":"2024-03-05T16:01:02.747243+0000","172.21.15.83:0/1025905465":"2024-03-05T16:01:02.747243+0000","172.21.15.83:6800/645025785":"2024-03-05T15:56:44.688018+0000","172.21.15.83:6801/1610714353":"2024-03-05T16:01:02.747243+0000","172.21.15.83:6800/1610714353":"2024-03-05T16:01:02.747243+0000","172.21.15.83:0/1017961836":"2024-03-05T15:56:44.688018+0000","172.21.15.83:0/1054338912":"2024-03-05T15:56:44.688018+0000","172.21.15.83:0/1893644918":"2024-03-05T16:01:02.747243+0000","172.21.15.83:6800/2606489355":"2024-03-05T15:56:22.880818+0000","172.21.15.83:6801/2606489355":"2024-03-05T15:56:22.880818+0000","172.21.15.83:6801/645025785":"2024-03-05T15:56:44.688018+0000","172.21.15.83:0/2983002978":"2024-03-05T15:56:22.880818+0000","172.21.15.83:0/489466195":"2024-03-05T15:56:44.688018+0000","172.21.15.83:0/1474787565":"2024-03-05T15:56:22.880818+0000","172.21.15.83:0/106501873":"2024-03-05T15:56:22.880818+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:01:25.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:25 smithi083 ceph-mon[24738]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:25.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:25 smithi083 ceph-mon[29452]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:26.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:25 smithi086 ceph-mon[25584]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:26.304 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 podman[44268]: 2024-03-04 16:01:26.088513175 +0000 UTC m=+3.108709682 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-04T16:01:26.305 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 podman[44268]: 2024-03-04 16:01:26.304276279 +0000 UTC m=+3.324472810 container create 7eb96ca65b539c2b0794cf3529a7c96782ddaeb04439ec299608bd0aecbc2422 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T16:01:26.607 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 podman[44268]: 2024-03-04 16:01:26.52279035 +0000 UTC m=+3.542986843 container init 7eb96ca65b539c2b0794cf3529a7c96782ddaeb04439ec299608bd0aecbc2422 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T16:01:26.607 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 podman[44268]: 2024-03-04 16:01:26.525495754 +0000 UTC m=+3.545692247 container start 7eb96ca65b539c2b0794cf3529a7c96782ddaeb04439ec299608bd0aecbc2422 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T16:01:26.607 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.529Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-04T16:01:26.607 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.529Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-04T16:01:26.607 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.529Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-04T16:01:26.607 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.529Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-04T16:01:26.607 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-04T16:01:26.608 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=arp 2024-03-04T16:01:26.608 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-04T16:01:26.608 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-04T16:01:26.608 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-04T16:01:26.608 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-04T16:01:26.608 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-04T16:01:26.608 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-04T16:01:26.608 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-04T16:01:26.608 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-04T16:01:26.608 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=edac 2024-03-04T16:01:26.609 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-04T16:01:26.609 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-04T16:01:26.609 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-04T16:01:26.609 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-04T16:01:26.609 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-04T16:01:26.609 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-04T16:01:26.609 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-04T16:01:26.609 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-04T16:01:26.609 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-04T16:01:26.609 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-04T16:01:26.609 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-04T16:01:26.610 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-04T16:01:26.610 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-04T16:01:26.610 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-04T16:01:26.610 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-04T16:01:26.610 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-04T16:01:26.610 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=os 2024-03-04T16:01:26.610 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-04T16:01:26.610 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-04T16:01:26.610 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-04T16:01:26.610 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-04T16:01:26.610 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-04T16:01:26.610 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-04T16:01:26.610 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=stat 2024-03-04T16:01:26.610 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-04T16:01:26.611 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-04T16:01:26.611 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-04T16:01:26.611 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=time 2024-03-04T16:01:26.611 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-04T16:01:26.611 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=uname 2024-03-04T16:01:26.611 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-04T16:01:26.611 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-04T16:01:26.611 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-04T16:01:26.611 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.530Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-04T16:01:26.611 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[44436]: ts=2024-03-04T16:01:26.531Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-04T16:01:26.611 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 bash[44268]: 7eb96ca65b539c2b0794cf3529a7c96782ddaeb04439ec299608bd0aecbc2422 2024-03-04T16:01:26.856 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:01:26 smithi083 systemd[1]: Started Ceph node-exporter.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:01:26.978 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:26 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/2004052599' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:01:26.978 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:26 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:27.127 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2004052599' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:01:27.127 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:27.127 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/2004052599' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:01:27.127 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:26 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:27.149 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-04T16:01:27.501 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T16:01:27.502 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:01:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:01:27.502 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:27 smithi086 systemd[1]: Starting Ceph node-exporter.b for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:01:27.719 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:01:27.858 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:27 smithi086 bash[40856]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-03-04T16:01:28.172 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:27 smithi083 ceph-mon[24738]: Deploying daemon node-exporter.b on smithi086 2024-03-04T16:01:28.172 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:27 smithi083 ceph-mon[24738]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:28.173 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:27 smithi083 ceph-mon[29452]: Deploying daemon node-exporter.b on smithi086 2024-03-04T16:01:28.173 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:27 smithi083 ceph-mon[29452]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:28.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:27 smithi086 ceph-mon[25584]: Deploying daemon node-exporter.b on smithi086 2024-03-04T16:01:28.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:27 smithi086 ceph-mon[25584]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:29.756 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-04T16:01:29.756 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":60,"fsid":"98ffad3c-da3f-11ee-95c6-87774f69a715","created":"2024-03-04T15:56:00.302465+0000","modified":"2024-03-04T16:01:02.747283+0000","last_up_change":"2024-03-04T16:00:47.205316+0000","last_in_change":"2024-03-04T16:00:30.390767+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T15:58:53.056819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"87dbf1ea-c0a9-4218-8659-1120adebf7e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":3013656475},{"type":"v1","addr":"172.21.15.83:6803","nonce":3013656475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":3013656475},{"type":"v1","addr":"172.21.15.83:6805","nonce":3013656475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3013656475},{"type":"v1","addr":"172.21.15.83:6809","nonce":3013656475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":3013656475},{"type":"v1","addr":"172.21.15.83:6807","nonce":3013656475}]},"public_addr":"172.21.15.83:6803/3013656475","cluster_addr":"172.21.15.83:6805/3013656475","heartbeat_back_addr":"172.21.15.83:6809/3013656475","heartbeat_front_addr":"172.21.15.83:6807/3013656475","state":["exists","up"]},{"osd":1,"uuid":"50c23dd2-2738-4b9b-8155-720d51c5a16c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2964268155},{"type":"v1","addr":"172.21.15.83:6811","nonce":2964268155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2964268155},{"type":"v1","addr":"172.21.15.83:6813","nonce":2964268155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2964268155},{"type":"v1","addr":"172.21.15.83:6817","nonce":2964268155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2964268155},{"type":"v1","addr":"172.21.15.83:6815","nonce":2964268155}]},"public_addr":"172.21.15.83:6811/2964268155","cluster_addr":"172.21.15.83:6813/2964268155","heartbeat_back_addr":"172.21.15.83:6817/2964268155","heartbeat_front_addr":"172.21.15.83:6815/2964268155","state":["exists","up"]},{"osd":2,"uuid":"6bf2ebd9-d898-49c4-befd-df999267dab5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1243652505},{"type":"v1","addr":"172.21.15.83:6819","nonce":1243652505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1243652505},{"type":"v1","addr":"172.21.15.83:6821","nonce":1243652505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":1243652505},{"type":"v1","addr":"172.21.15.83:6825","nonce":1243652505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1243652505},{"type":"v1","addr":"172.21.15.83:6823","nonce":1243652505}]},"public_addr":"172.21.15.83:6819/1243652505","cluster_addr":"172.21.15.83:6821/1243652505","heartbeat_back_addr":"172.21.15.83:6825/1243652505","heartbeat_front_addr":"172.21.15.83:6823/1243652505","state":["exists","up"]},{"osd":3,"uuid":"24508e11-465f-43e0-8aad-4ce41214b4b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1643785747},{"type":"v1","addr":"172.21.15.83:6827","nonce":1643785747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":1643785747},{"type":"v1","addr":"172.21.15.83:6829","nonce":1643785747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1643785747},{"type":"v1","addr":"172.21.15.83:6833","nonce":1643785747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":1643785747},{"type":"v1","addr":"172.21.15.83:6831","nonce":1643785747}]},"public_addr":"172.21.15.83:6827/1643785747","cluster_addr":"172.21.15.83:6829/1643785747","heartbeat_back_addr":"172.21.15.83:6833/1643785747","heartbeat_front_addr":"172.21.15.83:6831/1643785747","state":["exists","up"]},{"osd":4,"uuid":"79c9ab9a-0df7-485a-bba9-54470969695e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":745738669},{"type":"v1","addr":"172.21.15.86:6801","nonce":745738669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":745738669},{"type":"v1","addr":"172.21.15.86:6803","nonce":745738669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":745738669},{"type":"v1","addr":"172.21.15.86:6807","nonce":745738669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":745738669},{"type":"v1","addr":"172.21.15.86:6805","nonce":745738669}]},"public_addr":"172.21.15.86:6801/745738669","cluster_addr":"172.21.15.86:6803/745738669","heartbeat_back_addr":"172.21.15.86:6807/745738669","heartbeat_front_addr":"172.21.15.86:6805/745738669","state":["exists","up"]},{"osd":5,"uuid":"0e1dc2f0-4696-49d9-8691-109139cc3558","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":3306461240},{"type":"v1","addr":"172.21.15.86:6809","nonce":3306461240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":3306461240},{"type":"v1","addr":"172.21.15.86:6811","nonce":3306461240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":3306461240},{"type":"v1","addr":"172.21.15.86:6815","nonce":3306461240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":3306461240},{"type":"v1","addr":"172.21.15.86:6813","nonce":3306461240}]},"public_addr":"172.21.15.86:6809/3306461240","cluster_addr":"172.21.15.86:6811/3306461240","heartbeat_back_addr":"172.21.15.86:6815/3306461240","heartbeat_front_addr":"172.21.15.86:6813/3306461240","state":["exists","up"]},{"osd":6,"uuid":"bd5216b0-8187-40f2-a10f-724f8f8c95a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":3999097134},{"type":"v1","addr":"172.21.15.86:6817","nonce":3999097134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3999097134},{"type":"v1","addr":"172.21.15.86:6819","nonce":3999097134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3999097134},{"type":"v1","addr":"172.21.15.86:6823","nonce":3999097134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3999097134},{"type":"v1","addr":"172.21.15.86:6821","nonce":3999097134}]},"public_addr":"172.21.15.86:6817/3999097134","cluster_addr":"172.21.15.86:6819/3999097134","heartbeat_back_addr":"172.21.15.86:6823/3999097134","heartbeat_front_addr":"172.21.15.86:6821/3999097134","state":["exists","up"]},{"osd":7,"uuid":"11856cf1-448a-4a87-822a-4df57524fa47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":702851952},{"type":"v1","addr":"172.21.15.86:6825","nonce":702851952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":702851952},{"type":"v1","addr":"172.21.15.86:6827","nonce":702851952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":702851952},{"type":"v1","addr":"172.21.15.86:6831","nonce":702851952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":702851952},{"type":"v1","addr":"172.21.15.86:6829","nonce":702851952}]},"public_addr":"172.21.15.86:6825/702851952","cluster_addr":"172.21.15.86:6827/702851952","heartbeat_back_addr":"172.21.15.86:6831/702851952","heartbeat_front_addr":"172.21.15.86:6829/702851952","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:03.296875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:26.160457+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:58:49.263749+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:13.395869+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:35.202478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:59:59.251797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:00:22.154238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:00:45.756613+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:0/1947529438":"2024-03-05T16:01:02.747243+0000","172.21.15.83:0/3984588694":"2024-03-05T16:01:02.747243+0000","172.21.15.83:0/1025905465":"2024-03-05T16:01:02.747243+0000","172.21.15.83:6800/645025785":"2024-03-05T15:56:44.688018+0000","172.21.15.83:6801/1610714353":"2024-03-05T16:01:02.747243+0000","172.21.15.83:6800/1610714353":"2024-03-05T16:01:02.747243+0000","172.21.15.83:0/1017961836":"2024-03-05T15:56:44.688018+0000","172.21.15.83:0/1054338912":"2024-03-05T15:56:44.688018+0000","172.21.15.83:0/1893644918":"2024-03-05T16:01:02.747243+0000","172.21.15.83:6800/2606489355":"2024-03-05T15:56:22.880818+0000","172.21.15.83:6801/2606489355":"2024-03-05T15:56:22.880818+0000","172.21.15.83:6801/645025785":"2024-03-05T15:56:44.688018+0000","172.21.15.83:0/2983002978":"2024-03-05T15:56:22.880818+0000","172.21.15.83:0/489466195":"2024-03-05T15:56:44.688018+0000","172.21.15.83:0/1474787565":"2024-03-05T15:56:22.880818+0000","172.21.15.83:0/106501873":"2024-03-05T15:56:22.880818+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:01:29.861 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:29 smithi086 bash[40856]: Getting image source signatures 2024-03-04T16:01:29.861 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:29 smithi086 bash[40856]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-03-04T16:01:29.861 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:29 smithi086 bash[40856]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-03-04T16:01:29.861 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:29 smithi086 bash[40856]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-03-04T16:01:30.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:29 smithi083 ceph-mon[29452]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:30.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:29 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1405511429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:01:30.058 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:29 smithi083 ceph-mon[24738]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:30.058 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:29 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1405511429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:01:30.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:29 smithi086 ceph-mon[25584]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:30.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:29 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1405511429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:01:30.501 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-04T16:01:30.501 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-04T16:01:30.501 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-04T16:01:30.502 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-04T16:01:30.502 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-04T16:01:30.503 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-04T16:01:30.503 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-04T16:01:30.503 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-04T16:01:30.902 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:01:30.903 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:01:30.905 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:01:30.907 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:01:30.908 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:01:30.909 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:01:30.911 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:01:30.915 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:01:31.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-mon[25584]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:31.235 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:30 smithi086 bash[40856]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-03-04T16:01:31.235 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 bash[40856]: Writing manifest to image destination 2024-03-04T16:01:31.373 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:31 smithi083 ceph-mon[24738]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:31.374 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:31 smithi083 ceph-mon[29452]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:31.649 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 podman[40856]: 2024-03-04 16:01:31.314643579 +0000 UTC m=+3.864229582 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-04T16:01:31.649 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 podman[40856]: 2024-03-04 16:01:31.421661671 +0000 UTC m=+3.971247674 container create 9091c36c27a4a6270f0a45246698d1ea4342e0b6f4c325501f110c7daa425ea3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T16:01:31.649 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 podman[40856]: 2024-03-04 16:01:31.648688979 +0000 UTC m=+4.198274983 container init 9091c36c27a4a6270f0a45246698d1ea4342e0b6f4c325501f110c7daa425ea3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T16:01:31.985 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 podman[40856]: 2024-03-04 16:01:31.651348556 +0000 UTC m=+4.200934560 container start 9091c36c27a4a6270f0a45246698d1ea4342e0b6f4c325501f110c7daa425ea3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T16:01:31.985 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.654Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-04T16:01:31.985 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.654Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-04T16:01:31.985 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.655Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-04T16:01:31.985 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.655Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-04T16:01:31.985 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.655Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-04T16:01:31.985 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=arp 2024-03-04T16:01:31.985 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-04T16:01:31.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-04T16:01:31.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-04T16:01:31.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-04T16:01:31.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-04T16:01:31.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-04T16:01:31.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-04T16:01:31.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-04T16:01:31.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=edac 2024-03-04T16:01:31.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-04T16:01:31.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-04T16:01:31.987 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-04T16:01:31.987 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-04T16:01:31.987 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-04T16:01:31.987 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-04T16:01:31.987 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-04T16:01:31.987 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-04T16:01:31.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-04T16:01:31.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-04T16:01:31.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-04T16:01:31.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-04T16:01:31.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-04T16:01:31.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-04T16:01:31.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-04T16:01:31.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-04T16:01:31.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=os 2024-03-04T16:01:31.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-04T16:01:31.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-04T16:01:31.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-04T16:01:31.989 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-04T16:01:31.989 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-04T16:01:31.989 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-04T16:01:31.989 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=stat 2024-03-04T16:01:31.989 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-04T16:01:31.989 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-04T16:01:31.989 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-04T16:01:31.989 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=time 2024-03-04T16:01:31.989 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-04T16:01:31.989 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=uname 2024-03-04T16:01:31.989 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-04T16:01:31.989 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-04T16:01:31.990 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-04T16:01:31.990 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-04T16:01:31.990 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[40906]: ts=2024-03-04T16:01:31.656Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-04T16:01:31.990 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 bash[40856]: 9091c36c27a4a6270f0a45246698d1ea4342e0b6f4c325501f110c7daa425ea3 2024-03-04T16:01:31.990 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:01:31 smithi086 systemd[1]: Started Ceph node-exporter.b for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:01:32.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:32 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:32.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:32 smithi083 ceph-mon[24738]: Deploying daemon alertmanager.a on smithi083 2024-03-04T16:01:32.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:32 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:32.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:32 smithi083 ceph-mon[29452]: Deploying daemon alertmanager.a on smithi083 2024-03-04T16:01:33.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:32 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:33.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:32 smithi086 ceph-mon[25584]: Deploying daemon alertmanager.a on smithi083 2024-03-04T16:01:34.123 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:33 smithi083 ceph-mon[29452]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:34.124 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:33 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:34.124 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:33 smithi083 ceph-mon[24738]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:34.124 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:33 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:34.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:33 smithi086 ceph-mon[25584]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:34.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:33 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:01:35.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:35 smithi083 ceph-mon[24738]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:35.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:35 smithi083 ceph-mon[29452]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:35.989 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:35 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:01:35] "GET /metrics HTTP/1.1" 200 191470 "" "Prometheus/2.33.4" 2024-03-04T16:01:36.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:35 smithi086 ceph-mon[25584]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:37.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:37 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:01:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:01:37.938 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:37 smithi083 ceph-mon[24738]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:37.938 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:37 smithi083 ceph-mon[29452]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:38.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:37 smithi086 ceph-mon[25584]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:40.092 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:39 smithi083 ceph-mon[29452]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:40.092 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:39 smithi083 ceph-mon[24738]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:40.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:39 smithi086 ceph-mon[25584]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:41.187 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:41 smithi083 ceph-mon[29452]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:41.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:41 smithi083 ceph-mon[24738]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:41.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:41 smithi086 ceph-mon[25584]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:43.108 INFO:teuthology.orchestra.run.smithi083.stdout:236223201293 2024-03-04T16:01:43.109 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-04T16:01:43.868 INFO:teuthology.orchestra.run.smithi083.stdout:111669149727 2024-03-04T16:01:43.869 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-04T16:01:44.109 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:43 smithi083 ceph-mon[29452]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:44.109 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:43 smithi083 ceph-mon[24738]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:44.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:43 smithi086 ceph-mon[25584]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:45.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:45 smithi083 ceph-mon[24738]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:45.989 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:01:45] "GET /metrics HTTP/1.1" 200 191470 "" "Prometheus/2.33.4" 2024-03-04T16:01:45.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:45 smithi083 ceph-mon[29452]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:46.186 INFO:teuthology.orchestra.run.smithi083.stdout:34359738414 2024-03-04T16:01:46.186 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-04T16:01:46.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:45 smithi086 ceph-mon[25584]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:47.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:47 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:01:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:01:48.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:47 smithi083 ceph-mon[24738]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:48.209 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:47 smithi083 ceph-mon[29452]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:48.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:47 smithi086 ceph-mon[25584]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:48.323 INFO:teuthology.orchestra.run.smithi083.stdout:77309411366 2024-03-04T16:01:48.323 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-04T16:01:48.729 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:01:49.209 INFO:teuthology.orchestra.run.smithi083.stdout:167503724566 2024-03-04T16:01:49.210 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-04T16:01:50.003 INFO:teuthology.orchestra.run.smithi083.stdout:201863462929 2024-03-04T16:01:50.004 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-04T16:01:50.142 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:49 smithi083 ceph-mon[29452]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:50.142 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:49 smithi083 ceph-mon[24738]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:50.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:49 smithi086 ceph-mon[25584]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:50.755 INFO:teuthology.orchestra.run.smithi083.stdout:55834574889 2024-03-04T16:01:50.755 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-04T16:01:51.122 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:01:51.122 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:01:51.148 INFO:teuthology.orchestra.run.smithi083.stdout:133143986204 2024-03-04T16:01:51.149 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-04T16:01:51.367 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:51 smithi083 ceph-mon[24738]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:51.367 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:51 smithi083 ceph-mon[29452]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:51.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:51 smithi086 ceph-mon[25584]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:51.662 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:01:51.688 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:01:54.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:53 smithi083 ceph-mon[24738]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:54.219 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:53 smithi083 ceph-mon[29452]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:53 smithi086 ceph-mon[25584]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:56.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:55 smithi086 ceph-mon[25584]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:56.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:55 smithi083 ceph-mon[24738]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:56.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:55 smithi083 ceph-mon[29452]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:56.239 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:01:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:01:55] "GET /metrics HTTP/1.1" 200 191463 "" "Prometheus/2.33.4" 2024-03-04T16:01:56.926 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:01:56 smithi083 systemd[1]: Starting Ceph alertmanager.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:01:57.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:01:57 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:01:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:01:57.906 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:01:57.910 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:01:58.111 INFO:teuthology.orchestra.run.smithi083.stdout:236223201296 2024-03-04T16:01:58.111 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:57 smithi083 ceph-mon[24738]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:58.112 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:57 smithi083 ceph-mon[29452]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:57 smithi086 ceph-mon[25584]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:01:58.708 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:01:58.911 INFO:teuthology.orchestra.run.smithi083.stdout:201863462932 2024-03-04T16:01:58.911 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:58 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/416707858' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-04T16:01:58.911 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:58 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/416707858' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-04T16:01:59.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:58 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/416707858' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-04T16:02:00.013 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462929 got 201863462932 for osd.6 2024-03-04T16:02:00.013 DEBUG:teuthology.parallel:result is None 2024-03-04T16:02:00.129 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:59 smithi083 ceph-mon[24738]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:00.129 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:01:59 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1738187809' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-04T16:02:00.129 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:59 smithi083 ceph-mon[29452]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:00.130 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:01:59 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1738187809' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-04T16:02:00.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:59 smithi086 ceph-mon[25584]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:00.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:01:59 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1738187809' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-04T16:02:01.189 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:01 smithi083 ceph-mon[24738]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:01.189 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:01 smithi083 ceph-mon[29452]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:01.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:01 smithi086 ceph-mon[25584]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:01.565 INFO:teuthology.orchestra.run.smithi083.stdout:77309411369 2024-03-04T16:02:01.602 INFO:teuthology.orchestra.run.smithi083.stdout:111669149731 2024-03-04T16:02:02.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:02 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/873902584' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T16:02:02.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:02 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/4198167116' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-04T16:02:02.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:02 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/873902584' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T16:02:02.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:02 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/4198167116' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-04T16:02:02.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:02 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/873902584' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T16:02:02.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:02 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/4198167116' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-04T16:02:03.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:03 smithi086 ceph-mon[25584]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:03.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:02:03.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:03 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:02:03.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:03 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:02:03.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:03 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:02:03.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:03 smithi083 ceph-mon[24738]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:03.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:02:03.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:03 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:02:03.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:03 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:02:03.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:03 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:02:03.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:03 smithi083 ceph-mon[29452]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:03.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:02:03.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:03 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:02:03.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:03 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:02:03.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:03 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:02:04.607 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149727 got 111669149731 for osd.3 2024-03-04T16:02:04.607 DEBUG:teuthology.parallel:result is None 2024-03-04T16:02:05.408 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:05 smithi083 podman[46951]: 2024-03-04 16:02:04.963732805 +0000 UTC m=+0.148134388 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-04T16:02:05.408 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:05 smithi083 podman[46951]: 2024-03-04 16:02:05.103660437 +0000 UTC m=+0.288062041 volume create 317fab3c0708e7f8a9ff5586175b0099a3571a43da86a0f0ec7108fd5e727605 2024-03-04T16:02:05.408 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:05 smithi083 podman[46951]: 2024-03-04 16:02:05.212848262 +0000 UTC m=+0.397249824 container create e9d134cf60fb32a254019f7cbf42b1d944eb3945fab9d8bf7a0f5eb1ac26d8cc (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:02:05.906 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201293 got 236223201296 for osd.7 2024-03-04T16:02:05.907 DEBUG:teuthology.parallel:result is None 2024-03-04T16:02:06.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:05 smithi086 ceph-mon[25584]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:06.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:05 smithi083 ceph-mon[24738]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:06.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:05 smithi083 ceph-mon[29452]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:06.239 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:02:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:02:05] "GET /metrics HTTP/1.1" 200 191472 "" "Prometheus/2.33.4" 2024-03-04T16:02:06.267 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411366 got 77309411369 for osd.2 2024-03-04T16:02:06.267 DEBUG:teuthology.parallel:result is None 2024-03-04T16:02:06.508 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:06 smithi083 podman[46951]: 2024-03-04 16:02:06.346068306 +0000 UTC m=+1.530469888 container init e9d134cf60fb32a254019f7cbf42b1d944eb3945fab9d8bf7a0f5eb1ac26d8cc (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:02:06.508 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:06 smithi083 podman[46951]: 2024-03-04 16:02:06.349086223 +0000 UTC m=+1.533487788 container start e9d134cf60fb32a254019f7cbf42b1d944eb3945fab9d8bf7a0f5eb1ac26d8cc (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:02:06.509 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:06 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[46972]: level=info ts=2024-03-04T16:02:06.360Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-04T16:02:06.509 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:06 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[46972]: level=info ts=2024-03-04T16:02:06.360Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-04T16:02:06.509 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:06 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[46972]: level=info ts=2024-03-04T16:02:06.361Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.83 port=9094 2024-03-04T16:02:06.509 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:06 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[46972]: level=info ts=2024-03-04T16:02:06.362Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-04T16:02:06.509 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:06 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[46972]: level=info ts=2024-03-04T16:02:06.386Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-04T16:02:06.509 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:06 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[46972]: level=info ts=2024-03-04T16:02:06.387Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-04T16:02:06.509 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:06 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[46972]: level=info ts=2024-03-04T16:02:06.389Z caller=main.go:518 msg=Listening address=:9093 2024-03-04T16:02:06.509 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:06 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[46972]: level=info ts=2024-03-04T16:02:06.389Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-04T16:02:06.509 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:06 smithi083 bash[46951]: e9d134cf60fb32a254019f7cbf42b1d944eb3945fab9d8bf7a0f5eb1ac26d8cc 2024-03-04T16:02:06.509 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:06 smithi083 systemd[1]: Started Ceph alertmanager.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:02:07.443 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:02:07 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:02:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:02:07.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:07 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:07.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:07 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:07.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:07 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:07.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:07 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:07.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:07 smithi086 ceph-mon[25584]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:07.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:07 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3578354018' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-04T16:02:07.806 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:07 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:07.806 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:07 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:07.806 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:07 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:07.806 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:07 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:07.806 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:07 smithi083 ceph-mon[24738]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:07.806 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:07 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3578354018' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-04T16:02:07.807 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:07 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:07.807 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:07 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:07.807 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:07 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:07.807 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:07 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:07.807 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:07 smithi083 ceph-mon[29452]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:07.807 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:07 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3578354018' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-04T16:02:08.111 INFO:teuthology.orchestra.run.smithi083.stdout:34359738419 2024-03-04T16:02:08.738 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:08 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[46972]: level=info ts=2024-03-04T16:02:08.363Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001027156s 2024-03-04T16:02:08.795 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738414 got 34359738419 for osd.0 2024-03-04T16:02:08.795 DEBUG:teuthology.parallel:result is None 2024-03-04T16:02:08.998 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:08 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:08.999 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:08 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/2472600482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T16:02:09.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:08 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:09.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:08 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2472600482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T16:02:09.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:08 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:09.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:08 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/2472600482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T16:02:09.124 INFO:teuthology.orchestra.run.smithi083.stdout:167503724570 2024-03-04T16:02:09.734 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724566 got 167503724570 for osd.5 2024-03-04T16:02:09.734 DEBUG:teuthology.parallel:result is None 2024-03-04T16:02:09.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:09 smithi083 ceph-mon[29452]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:09.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:09 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:09.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:09 smithi083 ceph-mon[24738]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:09.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:09 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:10.121 INFO:teuthology.orchestra.run.smithi083.stdout:133143986208 2024-03-04T16:02:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:09 smithi086 ceph-mon[25584]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:09 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:10.427 INFO:teuthology.orchestra.run.smithi083.stdout:55834574894 2024-03-04T16:02:10.995 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574889 got 55834574894 for osd.1 2024-03-04T16:02:10.995 DEBUG:teuthology.parallel:result is None 2024-03-04T16:02:11.276 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:11 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/505173285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-04T16:02:11.276 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:11 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1558324171' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T16:02:11.276 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:11 smithi083 ceph-mon[29452]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:11.276 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:11 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/505173285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-04T16:02:11.277 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:11 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1558324171' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T16:02:11.277 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:11 smithi083 ceph-mon[24738]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:11.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:11 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/505173285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-04T16:02:11.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:11 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1558324171' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T16:02:11.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:11 smithi086 ceph-mon[25584]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:12.034 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986204 got 133143986208 for osd.4 2024-03-04T16:02:12.034 DEBUG:teuthology.parallel:result is None 2024-03-04T16:02:12.034 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-04T16:02:12.035 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:02:12.266 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:02:14.096 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-04T16:02:14.096 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:13 smithi083 ceph-mon[24738]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:14.097 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:13 smithi083 ceph-mon[29452]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:14.098 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-04T16:02:14.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:13 smithi086 ceph-mon[25584]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:15.459 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-03-04T16:02:12.810039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48904,"kb_used_data":6152,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912312,"statfs":{"total":767960285184,"available":767910207488,"internally_reserved":0,"allocated":6299648,"data_stored":3510946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002408"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":266,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-03-04T16:01:02.897330+0000","last_change":"2024-03-04T16:00:51.092873+0000","last_active":"2024-03-04T16:01:02.897330+0000","last_peered":"2024-03-04T16:01:02.897330+0000","last_clean":"2024-03-04T16:01:02.897330+0000","last_became_active":"2024-03-04T16:00:51.092504+0000","last_became_peered":"2024-03-04T16:00:51.092504+0000","last_unstale":"2024-03-04T16:01:02.897330+0000","last_undegraded":"2024-03-04T16:01:02.897330+0000","last_fullsized":"2024-03-04T16:01:02.897330+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:58:53.058483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:58:53.058483+0000","last_clean_scrub_stamp":"2024-03-04T15:58:53.058483+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:24:14.878453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201299,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 16:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.46500000000000002}]},{"osd":1,"last update":"Mon Mar 4 16:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.53300000000000003}]},{"osd":2,"last update":"Mon Mar 4 16:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.58999999999999997}]},{"osd":3,"last update":"Mon Mar 4 16:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.44800000000000001}]},{"osd":4,"last update":"Mon Mar 4 16:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.5},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.47799999999999998}]},{"osd":5,"last update":"Mon Mar 4 16:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.59999999999999998}]},{"osd":6,"last update":"Mon Mar 4 16:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.54500000000000004}]}]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":769000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.45800000000000002}]},{"osd":1,"last update":"Mon Mar 4 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.439},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.379}]},{"osd":2,"last update":"Mon Mar 4 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.42999999999999999}]},{"osd":3,"last update":"Mon Mar 4 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.45000000000000001}]},{"osd":4,"last update":"Mon Mar 4 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.41799999999999998}]},{"osd":5,"last update":"Mon Mar 4 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.33800000000000002}]},{"osd":7,"last update":"Mon Mar 4 16:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.313}]}]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6340,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 16:01:31 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.33600000000000002,"15min":0.32000000000000001},"min":{"1min":0.249,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.51700000000000002,"5min":0.627,"15min":0.627},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.34699999999999998,"15min":0.32600000000000001},"min":{"1min":0.26100000000000001,"5min":0.191,"15min":0.191},"max":{"1min":0.52800000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.377}]},{"osd":2,"last update":"Mon Mar 4 16:01:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36699999999999999,"15min":0.36499999999999999},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.48899999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.317},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.38900000000000001,"15min":0.38300000000000001},"min":{"1min":0.24299999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.50900000000000001,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.29699999999999999}]},{"osd":3,"last update":"Mon Mar 4 16:01:18 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.45700000000000002,"15min":0.46100000000000002},"min":{"1min":0.29899999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.439,"15min":0.442},"min":{"1min":0.314,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57999999999999996,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.33800000000000002}]},{"osd":4,"last update":"Mon Mar 4 16:01:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54200000000000004,"15min":0.54000000000000004},"min":{"1min":0.40799999999999997,"5min":0.379,"15min":0.379},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.44}]},{"osd":5,"last update":"Mon Mar 4 16:02:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.371,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.65100000000000002,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.59199999999999997,"15min":0.59599999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.78000000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.53800000000000003}]},{"osd":6,"last update":"Mon Mar 4 16:01:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.433},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.47799999999999998}]},{"osd":7,"last update":"Mon Mar 4 16:01:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.51700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 4 16:01:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.32300000000000001,"15min":0.29699999999999999},"min":{"1min":0.23899999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.33700000000000002,"15min":0.30599999999999999},"min":{"1min":0.22500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.53000000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.36299999999999999}]},{"osd":2,"last update":"Mon Mar 4 16:01:57 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.39600000000000002,"15min":0.38100000000000001},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.379,"15min":0.36799999999999999},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.372}]},{"osd":3,"last update":"Mon Mar 4 16:01:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.32700000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.433,"5min":0.44900000000000001,"15min":0.45100000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.28399999999999997}]},{"osd":4,"last update":"Mon Mar 4 16:01:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.86199999999999999,"15min":0.91400000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.98099999999999998,"5min":8.8230000000000004,"15min":8.8230000000000004},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.44700000000000001,"5min":0.442,"15min":0.442},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.38600000000000001}]},{"osd":5,"last update":"Mon Mar 4 16:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.57199999999999995,"15min":0.56699999999999995},"min":{"1min":0.40500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.44},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.59199999999999997,"15min":0.58499999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.40100000000000002}]},{"osd":6,"last update":"Mon Mar 4 16:01:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.46700000000000003}]},{"osd":7,"last update":"Mon Mar 4 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.47999999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6280,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.35899999999999999,"15min":0.35699999999999998},"min":{"1min":0.22500000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.48299999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.375,"5min":0.36799999999999999,"15min":0.36199999999999999},"min":{"1min":0.23000000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.55000000000000004,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.438}]},{"osd":1,"last update":"Mon Mar 4 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39100000000000001,"15min":0.38700000000000001},"min":{"1min":0.28799999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.54600000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.44,"15min":0.441},"min":{"1min":0.29599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.56200000000000006,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.40300000000000002}]},{"osd":3,"last update":"Mon Mar 4 16:01:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.53400000000000003,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.432,"15min":0.438},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.35599999999999998}]},{"osd":4,"last update":"Mon Mar 4 16:01:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.51300000000000001}]},{"osd":5,"last update":"Mon Mar 4 16:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.753,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.319,"5min":0.70499999999999996,"15min":0.60299999999999998},"min":{"1min":0.434,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":13.336,"5min":13.336,"15min":13.336},"last":0.52500000000000002}]},{"osd":6,"last update":"Mon Mar 4 16:01:25 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.47099999999999997}]},{"osd":7,"last update":"Mon Mar 4 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.57399999999999995}]}]},{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 16:01:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.39400000000000002,"15min":0.39100000000000001},"min":{"1min":0.24399999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60499999999999998,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.309},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.39500000000000002,"15min":0.38600000000000001},"min":{"1min":0.28199999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.36599999999999999}]},{"osd":1,"last update":"Mon Mar 4 16:01:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.42999999999999999,"15min":0.42599999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.42699999999999999,"15min":0.42399999999999999},"min":{"1min":0.29199999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64700000000000002,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.379}]},{"osd":2,"last update":"Mon Mar 4 16:01:19 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.46300000000000002,"15min":0.45700000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.44500000000000001,"15min":0.438},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.34899999999999998}]},{"osd":4,"last update":"Mon Mar 4 16:01:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.54400000000000004,"15min":0.53800000000000003},"min":{"1min":0.41899999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54200000000000004,"15min":0.54000000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.52500000000000002}]},{"osd":5,"last update":"Mon Mar 4 16:02:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.58199999999999996,"15min":0.58399999999999996},"min":{"1min":0.41499999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.441}]},{"osd":6,"last update":"Mon Mar 4 16:01:25 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.48999999999999999}]},{"osd":7,"last update":"Mon Mar 4 16:01:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.56799999999999995}]}]},{"osd":4,"up_from":31,"seq":133143986209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 16:01:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53100000000000003,"15min":0.53300000000000003},"min":{"1min":0.38700000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.74399999999999999,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.54700000000000004,"15min":0.55000000000000004},"min":{"1min":0.45000000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.624,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.625}]},{"osd":1,"last update":"Mon Mar 4 16:01:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53500000000000003,"15min":0.53300000000000003},"min":{"1min":0.41699999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72099999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.39100000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73299999999999998,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.58399999999999996}]},{"osd":2,"last update":"Mon Mar 4 16:01:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.55800000000000005,"15min":0.56299999999999994},"min":{"1min":0.39200000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.70099999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53500000000000003,"15min":0.53600000000000003},"min":{"1min":0.42499999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.67500000000000004,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.46600000000000003}]},{"osd":3,"last update":"Mon Mar 4 16:01:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.72999999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55100000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.66900000000000004,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.55000000000000004}]},{"osd":5,"last update":"Mon Mar 4 16:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.40600000000000003,"15min":0.40000000000000002},"min":{"1min":0.32700000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.623,"5min":0.625,"15min":0.625},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.435,"15min":0.42499999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.40000000000000002}]},{"osd":6,"last update":"Mon Mar 4 16:01:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.36499999999999999}]},{"osd":7,"last update":"Mon Mar 4 16:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.39400000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724571,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5636,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739516,"statfs":{"total":95995035648,"available":95989264384,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 16:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.49299999999999999,"15min":0.48499999999999999},"min":{"1min":0.41899999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.312}]},{"osd":1,"last update":"Mon Mar 4 16:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.501,"15min":0.48899999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.53300000000000003,"15min":0.53000000000000003},"min":{"1min":0.35799999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.45000000000000001}]},{"osd":2,"last update":"Mon Mar 4 16:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.51400000000000001,"15min":0.50800000000000001},"min":{"1min":0.376,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.52600000000000002,"15min":0.51800000000000002},"min":{"1min":0.44800000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.36899999999999999}]},{"osd":3,"last update":"Mon Mar 4 16:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.54900000000000004,"15min":0.54400000000000004},"min":{"1min":0.432,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54400000000000004,"15min":0.53800000000000003},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.379}]},{"osd":4,"last update":"Mon Mar 4 16:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.38800000000000001,"15min":0.372},"min":{"1min":0.28100000000000003,"5min":0.253,"15min":0.253},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.40000000000000002,"15min":0.378},"min":{"1min":0.34300000000000003,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.58099999999999996}]},{"osd":6,"last update":"Mon Mar 4 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.49399999999999999}]},{"osd":7,"last update":"Mon Mar 4 16:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.48399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:02:15.465 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:02:15.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:15 smithi083 ceph-mon[24738]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:02:15.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:15 smithi083 ceph-mon[24738]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:15.989 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:02:15 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:02:15] "GET /metrics HTTP/1.1" 200 191472 "" "Prometheus/2.33.4" 2024-03-04T16:02:15.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:15 smithi083 ceph-mon[29452]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:02:15.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:15 smithi083 ceph-mon[29452]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:16.149 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:02:16.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:15 smithi086 ceph-mon[25584]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:02:16.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:15 smithi086 ceph-mon[25584]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:16.481 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:16 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[46972]: level=info ts=2024-03-04T16:02:16.366Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003832895s 2024-03-04T16:02:17.472 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:17 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:17.472 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:17 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:17.472 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:17 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:17.472 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:17 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:17.472 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:17 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T16:02:17.472 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:17 smithi083 ceph-mon[29452]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T16:02:17.472 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:17 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:17.472 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:17 smithi083 ceph-mon[29452]: Deploying daemon grafana.a on smithi086 2024-03-04T16:02:17.472 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:17 smithi083 ceph-mon[29452]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:17.473 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:17 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:17.473 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:17 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:17.473 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:17 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:17.473 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:17 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:17.473 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:17 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T16:02:17.473 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:17 smithi083 ceph-mon[24738]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T16:02:17.473 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:17 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:17.473 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:17 smithi083 ceph-mon[24738]: Deploying daemon grafana.a on smithi086 2024-03-04T16:02:17.473 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:17 smithi083 ceph-mon[24738]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:17.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:02:17 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:02:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:02:17.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:17 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:17.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:17 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:17.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:17 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:17.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:17 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:17.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:17 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T16:02:17.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:17 smithi086 ceph-mon[25584]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T16:02:17.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:17 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:17.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:17 smithi086 ceph-mon[25584]: Deploying daemon grafana.a on smithi086 2024-03-04T16:02:17.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:17 smithi086 ceph-mon[25584]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:17.675 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-04T16:02:17.677 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-04T16:02:18.281 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-03-04T16:02:16.810667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48904,"kb_used_data":6152,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912312,"statfs":{"total":767960285184,"available":767910207488,"internally_reserved":0,"allocated":6299648,"data_stored":3510946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002144"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":266,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-03-04T16:01:02.897330+0000","last_change":"2024-03-04T16:00:51.092873+0000","last_active":"2024-03-04T16:01:02.897330+0000","last_peered":"2024-03-04T16:01:02.897330+0000","last_clean":"2024-03-04T16:01:02.897330+0000","last_became_active":"2024-03-04T16:00:51.092504+0000","last_became_peered":"2024-03-04T16:00:51.092504+0000","last_unstale":"2024-03-04T16:01:02.897330+0000","last_undegraded":"2024-03-04T16:01:02.897330+0000","last_fullsized":"2024-03-04T16:01:02.897330+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:58:53.058483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:58:53.058483+0000","last_clean_scrub_stamp":"2024-03-04T15:58:53.058483+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:24:14.878453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201300,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 16:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.44}]},{"osd":1,"last update":"Mon Mar 4 16:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.56100000000000005}]},{"osd":2,"last update":"Mon Mar 4 16:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.61699999999999999}]},{"osd":3,"last update":"Mon Mar 4 16:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.58399999999999996}]},{"osd":4,"last update":"Mon Mar 4 16:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.45400000000000001}]},{"osd":5,"last update":"Mon Mar 4 16:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.504},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.52200000000000002}]},{"osd":6,"last update":"Mon Mar 4 16:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.626},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.54000000000000004}]}]},{"osd":6,"up_from":47,"seq":201863462936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":769000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.46600000000000003}]},{"osd":1,"last update":"Mon Mar 4 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.41699999999999998}]},{"osd":2,"last update":"Mon Mar 4 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.34300000000000003}]},{"osd":3,"last update":"Mon Mar 4 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.504},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.53100000000000003}]},{"osd":4,"last update":"Mon Mar 4 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Mar 4 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.58999999999999997}]},{"osd":7,"last update":"Mon Mar 4 16:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.60499999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6340,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 16:01:31 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.33600000000000002,"15min":0.32000000000000001},"min":{"1min":0.249,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.51700000000000002,"5min":0.627,"15min":0.627},"last":0.307},{"interface":"front","average":{"1min":0.377,"5min":0.34699999999999998,"15min":0.32600000000000001},"min":{"1min":0.26100000000000001,"5min":0.191,"15min":0.191},"max":{"1min":0.52800000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.32500000000000001}]},{"osd":2,"last update":"Mon Mar 4 16:01:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36699999999999999,"15min":0.36499999999999999},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.48899999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.38900000000000001,"15min":0.38300000000000001},"min":{"1min":0.24299999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.50900000000000001,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.29199999999999998}]},{"osd":3,"last update":"Mon Mar 4 16:01:18 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.45700000000000002,"15min":0.46100000000000002},"min":{"1min":0.29899999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.248},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.439,"15min":0.442},"min":{"1min":0.314,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57999999999999996,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.25800000000000001}]},{"osd":4,"last update":"Mon Mar 4 16:01:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54200000000000004,"15min":0.54000000000000004},"min":{"1min":0.40799999999999997,"5min":0.379,"15min":0.379},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.54800000000000004}]},{"osd":5,"last update":"Mon Mar 4 16:02:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.371,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.65100000000000002,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.59199999999999997,"15min":0.59599999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.78000000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.56699999999999995}]},{"osd":6,"last update":"Mon Mar 4 16:01:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.64700000000000002}]},{"osd":7,"last update":"Mon Mar 4 16:01:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.63},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.57699999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 4 16:01:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.32300000000000001,"15min":0.29699999999999999},"min":{"1min":0.23899999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.33700000000000002,"15min":0.30599999999999999},"min":{"1min":0.22500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.53000000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.309}]},{"osd":2,"last update":"Mon Mar 4 16:01:57 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.39600000000000002,"15min":0.38100000000000001},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.379,"15min":0.36799999999999999},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.28799999999999998}]},{"osd":3,"last update":"Mon Mar 4 16:01:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.32700000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.433,"5min":0.44900000000000001,"15min":0.45100000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.31900000000000001}]},{"osd":4,"last update":"Mon Mar 4 16:01:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.86199999999999999,"15min":0.91400000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.98099999999999998,"5min":8.8230000000000004,"15min":8.8230000000000004},"last":0.371},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.44700000000000001,"5min":0.442,"15min":0.442},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.39600000000000002}]},{"osd":5,"last update":"Mon Mar 4 16:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.57199999999999995,"15min":0.56699999999999995},"min":{"1min":0.40500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.59199999999999997,"15min":0.58499999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.38}]},{"osd":6,"last update":"Mon Mar 4 16:01:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.41999999999999998}]},{"osd":7,"last update":"Mon Mar 4 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.495}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6280,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.35899999999999999,"15min":0.35699999999999998},"min":{"1min":0.22500000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.48299999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.251},{"interface":"front","average":{"1min":0.375,"5min":0.36799999999999999,"15min":0.36199999999999999},"min":{"1min":0.23000000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.55000000000000004,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.35699999999999998}]},{"osd":1,"last update":"Mon Mar 4 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39100000000000001,"15min":0.38700000000000001},"min":{"1min":0.28799999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.54600000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.44,"15min":0.441},"min":{"1min":0.29599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.56200000000000006,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.314}]},{"osd":3,"last update":"Mon Mar 4 16:01:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.53400000000000003,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.432,"15min":0.438},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.34599999999999997}]},{"osd":4,"last update":"Mon Mar 4 16:01:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.68899999999999995}]},{"osd":5,"last update":"Mon Mar 4 16:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.753,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.319,"5min":0.70499999999999996,"15min":0.60299999999999998},"min":{"1min":0.434,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":13.336,"5min":13.336,"15min":13.336},"last":0.57199999999999995}]},{"osd":6,"last update":"Mon Mar 4 16:01:25 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.64100000000000001}]},{"osd":7,"last update":"Mon Mar 4 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.65300000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 16:01:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.39400000000000002,"15min":0.39100000000000001},"min":{"1min":0.24399999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60499999999999998,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.39500000000000002,"15min":0.38600000000000001},"min":{"1min":0.28199999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.32900000000000001}]},{"osd":1,"last update":"Mon Mar 4 16:01:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.42999999999999999,"15min":0.42599999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.42699999999999999,"15min":0.42399999999999999},"min":{"1min":0.29199999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64700000000000002,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.29899999999999999}]},{"osd":2,"last update":"Mon Mar 4 16:01:19 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.46300000000000002,"15min":0.45700000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.44500000000000001,"15min":0.438},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.39700000000000002}]},{"osd":4,"last update":"Mon Mar 4 16:01:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.54400000000000004,"15min":0.53800000000000003},"min":{"1min":0.41899999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54200000000000004,"15min":0.54000000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.56899999999999995}]},{"osd":5,"last update":"Mon Mar 4 16:02:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.58199999999999996,"15min":0.58399999999999996},"min":{"1min":0.41499999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.51800000000000002}]},{"osd":6,"last update":"Mon Mar 4 16:01:25 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.59199999999999997}]},{"osd":7,"last update":"Mon Mar 4 16:01:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.50600000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986210,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 16:01:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53100000000000003,"15min":0.53300000000000003},"min":{"1min":0.38700000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.74399999999999999,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.54700000000000004,"15min":0.55000000000000004},"min":{"1min":0.45000000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.624,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.66900000000000004}]},{"osd":1,"last update":"Mon Mar 4 16:01:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53500000000000003,"15min":0.53300000000000003},"min":{"1min":0.41699999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72099999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.39100000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73299999999999998,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.68899999999999995}]},{"osd":2,"last update":"Mon Mar 4 16:01:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.55800000000000005,"15min":0.56299999999999994},"min":{"1min":0.39200000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.70099999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53500000000000003,"15min":0.53600000000000003},"min":{"1min":0.42499999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.67500000000000004,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.58699999999999997}]},{"osd":3,"last update":"Mon Mar 4 16:01:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.72999999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.624},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55100000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.66900000000000004,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.61699999999999999}]},{"osd":5,"last update":"Mon Mar 4 16:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.40600000000000003,"15min":0.40000000000000002},"min":{"1min":0.32700000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.623,"5min":0.625,"15min":0.625},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.435,"15min":0.42499999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.46800000000000003}]},{"osd":6,"last update":"Mon Mar 4 16:01:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.48899999999999999}]},{"osd":7,"last update":"Mon Mar 4 16:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.501}]}]},{"osd":5,"up_from":39,"seq":167503724572,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5636,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739516,"statfs":{"total":95995035648,"available":95989264384,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 4 16:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.49299999999999999,"15min":0.48499999999999999},"min":{"1min":0.41899999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.58099999999999996}]},{"osd":1,"last update":"Mon Mar 4 16:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.501,"15min":0.48899999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.53300000000000003,"15min":0.53000000000000003},"min":{"1min":0.35799999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.60699999999999998}]},{"osd":2,"last update":"Mon Mar 4 16:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.51400000000000001,"15min":0.50800000000000001},"min":{"1min":0.376,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.52600000000000002,"15min":0.51800000000000002},"min":{"1min":0.44800000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Mar 4 16:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.54900000000000004,"15min":0.54400000000000004},"min":{"1min":0.432,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54400000000000004,"15min":0.53800000000000003},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.48599999999999999}]},{"osd":4,"last update":"Mon Mar 4 16:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.38800000000000001,"15min":0.372},"min":{"1min":0.28100000000000003,"5min":0.253,"15min":0.253},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.377},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.40000000000000002,"15min":0.378},"min":{"1min":0.34300000000000003,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.46700000000000003}]},{"osd":6,"last update":"Mon Mar 4 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.59499999999999997}]},{"osd":7,"last update":"Mon Mar 4 16:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.45600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:02:18.282 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-04T16:02:18.283 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-04T16:02:18.283 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-04T16:02:18.283 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- ceph health --format=json 2024-03-04T16:02:18.496 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:02:18.838 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:18 smithi083 ceph-mon[29452]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:02:18.838 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:18 smithi083 ceph-mon[24738]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:02:18.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:18 smithi086 ceph-mon[25584]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:02:19.887 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:19 smithi083 ceph-mon[24738]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:19.887 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:19 smithi083 ceph-mon[29452]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:19.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:19 smithi086 ceph-mon[25584]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:20.317 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-04T16:02:20.317 INFO:teuthology.orchestra.run.smithi083.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-04T16:02:20.937 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-04T16:02:20.938 INFO:tasks.cephadm:Setup complete, yielding 2024-03-04T16:02:20.938 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-04T16:02:20.947 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi083.front.sepia.ceph.com 2024-03-04T16:02:20.948 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-03-04T16:02:21.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:21 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1310793472' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T16:02:21.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:21 smithi083 ceph-mon[24738]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:21.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:21 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1310793472' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T16:02:21.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:21 smithi083 ceph-mon[29452]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:21.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:21 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1310793472' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T16:02:21.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:21 smithi086 ceph-mon[25584]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:23.543 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-04T16:02:23.552 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi083.front.sepia.ceph.com 2024-03-04T16:02:23.552 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-03-04T16:02:24.174 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:23 smithi083 ceph-mon[24738]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:24.174 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:23 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/270444007' entity='client.admin' 2024-03-04T16:02:24.174 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:23 smithi083 ceph-mon[29452]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:24.175 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:23 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/270444007' entity='client.admin' 2024-03-04T16:02:24.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:23 smithi086 ceph-mon[25584]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:24.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:23 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/270444007' entity='client.admin' 2024-03-04T16:02:26.215 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:02:25 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:02:25] "GET /metrics HTTP/1.1" 200 191477 "" "Prometheus/2.33.4" 2024-03-04T16:02:26.215 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:25 smithi083 ceph-mon[24738]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:26.216 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:25 smithi083 ceph-mon[29452]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:26.228 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:25 smithi086 ceph-mon[25584]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:27.146 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:02:27.147 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "dbdacf3e-1b44-4085-8338-e694476a974b", 2024-03-04T16:02:27.147 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "r", 2024-03-04T16:02:27.147 INFO:teuthology.orchestra.run.smithi083.stdout: "current_period": "268b3a83-baa8-4f22-badb-7330b2a001b7", 2024-03-04T16:02:27.147 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1 2024-03-04T16:02:27.147 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:02:27.467 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:27 smithi086 ceph-mon[25584]: osdmap e61: 8 total, 8 up, 8 in 2024-03-04T16:02:27.467 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:27 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/4011725737' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-04T16:02:27.467 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:27 smithi086 ceph-mon[25584]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:27.468 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:02:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:02:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:02:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:27 smithi083 ceph-mon[29452]: osdmap e61: 8 total, 8 up, 8 in 2024-03-04T16:02:27.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:27 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/4011725737' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-04T16:02:27.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:27 smithi083 ceph-mon[29452]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:27 smithi083 ceph-mon[24738]: osdmap e61: 8 total, 8 up, 8 in 2024-03-04T16:02:27.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:27 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/4011725737' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-04T16:02:27.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:27 smithi083 ceph-mon[24738]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:27.859 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-03-04T16:02:28.372 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:28 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/4011725737' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-04T16:02:28.372 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:28 smithi086 ceph-mon[25584]: osdmap e62: 8 total, 8 up, 8 in 2024-03-04T16:02:28.463 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:28 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/4011725737' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-04T16:02:28.463 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:28 smithi083 ceph-mon[29452]: osdmap e62: 8 total, 8 up, 8 in 2024-03-04T16:02:28.463 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:28 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/4011725737' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-04T16:02:28.464 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:28 smithi083 ceph-mon[24738]: osdmap e62: 8 total, 8 up, 8 in 2024-03-04T16:02:28.734 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:28 smithi086 systemd[1]: Starting Ceph grafana.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:02:29.100 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:28 smithi086 podman[41668]: 2024-03-04 16:02:28.705168965 +0000 UTC m=+0.016162079 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-04T16:02:29.100 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:28 smithi086 podman[41668]: 2024-03-04 16:02:28.822983806 +0000 UTC m=+0.133976950 container create 9c0f7c6afe4b9a92ad01e1972af61608d582d1ceb7879bc67c97f163c8953a53 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a, version=8.5, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, vcs-type=git, build-date=2022-03-28T10:36:18.413762, name=ubi8, distribution-scope=public, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, description=Ceph Grafana Container) 2024-03-04T16:02:29.100 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 podman[41668]: 2024-03-04 16:02:29.100275114 +0000 UTC m=+0.411268218 container init 9c0f7c6afe4b9a92ad01e1972af61608d582d1ceb7879bc67c97f163c8953a53 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, architecture=x86_64, io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, maintainer=Paul Cuzner , distribution-scope=public) 2024-03-04T16:02:29.350 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:29 smithi083 ceph-mon[29452]: osdmap e63: 8 total, 8 up, 8 in 2024-03-04T16:02:29.469 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:29 smithi083 ceph-mon[29452]: pgmap v49: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:29.470 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:29 smithi083 ceph-mon[24738]: osdmap e63: 8 total, 8 up, 8 in 2024-03-04T16:02:29.470 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:29 smithi083 ceph-mon[24738]: pgmap v49: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:29.470 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-mon[25584]: osdmap e63: 8 total, 8 up, 8 in 2024-03-04T16:02:29.470 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-mon[25584]: pgmap v49: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:29.471 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 podman[41668]: 2024-03-04 16:02:29.103110313 +0000 UTC m=+0.414103426 container start 9c0f7c6afe4b9a92ad01e1972af61608d582d1ceb7879bc67c97f163c8953a53 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, name=ubi8, maintainer=Paul Cuzner , release=236.1648460182, io.openshift.tags=base rhel8, io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, vcs-type=git, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container) 2024-03-04T16:02:29.471 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-03-04T16:02:29.472 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-03-04T16:02:29.472 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-03-04T16:02:29.472 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-03-04T16:02:29.472 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-04T16:02:29.472 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-04T16:02:29.472 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-04T16:02:29.472 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-04T16:02:29.472 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-03-04T16:02:29.472 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-03-04T16:02:29.472 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-03-04T16:02:29.472 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-03-04T16:02:29.472 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-03-04T16:02:29.472 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="App mode production" logger=settings 2024-03-04T16:02:29.473 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-03-04T16:02:29.473 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-04T16:02:29.473 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-03-04T16:02:29.473 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-03-04T16:02:29.473 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-03-04T16:02:29.473 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-03-04T16:02:29.473 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-03-04T16:02:29.473 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-03-04T16:02:29.473 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-03-04T16:02:29.473 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-03-04T16:02:29.473 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-03-04T16:02:29.473 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-03-04T16:02:29.473 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-03-04T16:02:29.473 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-03-04T16:02:29.474 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-03-04T16:02:29.474 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-03-04T16:02:29.474 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-03-04T16:02:29.474 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-03-04T16:02:29.474 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-03-04T16:02:29.474 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-03-04T16:02:29.474 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-03-04T16:02:29.474 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-03-04T16:02:29.474 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-03-04T16:02:29.474 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-03-04T16:02:29.474 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-03-04T16:02:29.474 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-03-04T16:02:29.474 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-03-04T16:02:29.474 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-03-04T16:02:29.475 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-03-04T16:02:29.475 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-03-04T16:02:29.475 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-03-04T16:02:29.475 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-03-04T16:02:29.475 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-04T16:02:29.475 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-03-04T16:02:29.475 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-03-04T16:02:29.475 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 bash[41668]: 9c0f7c6afe4b9a92ad01e1972af61608d582d1ceb7879bc67c97f163c8953a53 2024-03-04T16:02:29.475 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-03-04T16:02:29.475 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-03-04T16:02:29.475 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 systemd[1]: Started Ceph grafana.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:02:29.475 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-03-04T16:02:29.475 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-03-04T16:02:29.475 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-03-04T16:02:29.475 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-03-04T16:02:29.476 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-03-04T16:02:29.476 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-03-04T16:02:29.476 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-03-04T16:02:29.476 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-03-04T16:02:29.476 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-03-04T16:02:29.476 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-03-04T16:02:29.476 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-03-04T16:02:29.476 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-03-04T16:02:29.476 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-03-04T16:02:29.476 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-03-04T16:02:29.476 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-03-04T16:02:29.476 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-03-04T16:02:29.477 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-03-04T16:02:29.477 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-03-04T16:02:29.477 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-03-04T16:02:29.477 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-03-04T16:02:29.477 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-04T16:02:29.477 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-03-04T16:02:29.477 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-03-04T16:02:29.477 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-03-04T16:02:29.477 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-03-04T16:02:29.477 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-03-04T16:02:29.477 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-03-04T16:02:29.477 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-03-04T16:02:29.477 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-03-04T16:02:29.478 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-03-04T16:02:29.478 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-03-04T16:02:29.478 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-03-04T16:02:29.478 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-03-04T16:02:29.478 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-03-04T16:02:29.478 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-03-04T16:02:29.478 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-03-04T16:02:29.479 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-03-04T16:02:29.479 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-03-04T16:02:29.479 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-03-04T16:02:29.479 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-03-04T16:02:29.479 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-03-04T16:02:29.479 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-03-04T16:02:29.479 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-03-04T16:02:29.479 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-03-04T16:02:29.479 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-03-04T16:02:29.479 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-04T16:02:29.479 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-03-04T16:02:29.479 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-04T16:02:29.479 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-03-04T16:02:29.479 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-04T16:02:29.480 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-04T16:02:29.480 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-03-04T16:02:29.480 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-03-04T16:02:29.480 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-03-04T16:02:29.480 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-03-04T16:02:29.480 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-03-04T16:02:29.480 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-03-04T16:02:29.480 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-03-04T16:02:29.480 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-03-04T16:02:29.480 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-03-04T16:02:29.480 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-03-04T16:02:29.480 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-03-04T16:02:29.481 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-03-04T16:02:29.481 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-03-04T16:02:29.481 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-03-04T16:02:29.481 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-03-04T16:02:29.481 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-03-04T16:02:29.481 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-03-04T16:02:29.481 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-03-04T16:02:29.481 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-03-04T16:02:29.481 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-03-04T16:02:29.481 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-03-04T16:02:29.481 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-03-04T16:02:29.481 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-03-04T16:02:29.481 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-03-04T16:02:29.481 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-03-04T16:02:29.481 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-03-04T16:02:29.482 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-03-04T16:02:29.482 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-03-04T16:02:29.482 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-03-04T16:02:29.482 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-03-04T16:02:29.482 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-03-04T16:02:29.482 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-03-04T16:02:29.482 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-03-04T16:02:29.482 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-03-04T16:02:29.482 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-03-04T16:02:29.482 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-03-04T16:02:29.482 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-03-04T16:02:29.482 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-03-04T16:02:29.482 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-03-04T16:02:29.482 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-03-04T16:02:29.483 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-03-04T16:02:29.483 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-03-04T16:02:29.483 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-03-04T16:02:29.483 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-03-04T16:02:29.483 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-03-04T16:02:29.483 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-03-04T16:02:29.483 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-03-04T16:02:29.483 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-03-04T16:02:29.483 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-03-04T16:02:29.579 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:02:29.579 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "c0f800a7-05b5-44df-ba36-43c7bf5fbcbb", 2024-03-04T16:02:29.579 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "default", 2024-03-04T16:02:29.579 INFO:teuthology.orchestra.run.smithi083.stdout: "api_name": "default", 2024-03-04T16:02:29.579 INFO:teuthology.orchestra.run.smithi083.stdout: "is_master": "true", 2024-03-04T16:02:29.579 INFO:teuthology.orchestra.run.smithi083.stdout: "endpoints": [], 2024-03-04T16:02:29.579 INFO:teuthology.orchestra.run.smithi083.stdout: "hostnames": [], 2024-03-04T16:02:29.579 INFO:teuthology.orchestra.run.smithi083.stdout: "hostnames_s3website": [], 2024-03-04T16:02:29.579 INFO:teuthology.orchestra.run.smithi083.stdout: "master_zone": "", 2024-03-04T16:02:29.579 INFO:teuthology.orchestra.run.smithi083.stdout: "zones": [], 2024-03-04T16:02:29.579 INFO:teuthology.orchestra.run.smithi083.stdout: "placement_targets": [], 2024-03-04T16:02:29.580 INFO:teuthology.orchestra.run.smithi083.stdout: "default_placement": "", 2024-03-04T16:02:29.580 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_id": "dbdacf3e-1b44-4085-8338-e694476a974b", 2024-03-04T16:02:29.580 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_policy": { 2024-03-04T16:02:29.580 INFO:teuthology.orchestra.run.smithi083.stdout: "groups": [] 2024-03-04T16:02:29.580 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:02:29.580 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:02:29.602 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-03-04T16:02:29.602 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-04T16:02:29.602 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-04T16:02:29.602 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-03-04T16:02:29.602 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-03-04T16:02:29.602 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-03-04T16:02:29.602 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-03-04T16:02:29.602 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-04T16:02:29.602 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-03-04T16:02:29.603 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-04T16:02:29.603 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-03-04T16:02:29.603 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-03-04T16:02:29.603 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-04T16:02:29.603 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-03-04T16:02:29.603 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-03-04T16:02:29.603 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-03-04T16:02:29.603 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-03-04T16:02:29.603 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-03-04T16:02:29.603 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-03-04T16:02:29.604 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-03-04T16:02:29.604 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-03-04T16:02:29.604 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-03-04T16:02:29.605 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-03-04T16:02:29.605 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-03-04T16:02:29.605 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-03-04T16:02:29.605 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-03-04T16:02:29.605 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-03-04T16:02:29.605 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-03-04T16:02:29.605 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-03-04T16:02:29.605 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-03-04T16:02:29.605 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-03-04T16:02:29.605 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-03-04T16:02:29.605 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-03-04T16:02:29.605 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-03-04T16:02:29.605 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-04T16:02:29.605 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-04T16:02:29.606 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-04T16:02:29.606 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-03-04T16:02:29.606 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-04T16:02:29.606 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-03-04T16:02:29.606 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-03-04T16:02:29.606 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-03-04T16:02:29.606 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-03-04T16:02:29.606 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-03-04T16:02:29.606 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-03-04T16:02:29.606 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-03-04T16:02:29.606 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-03-04T16:02:29.606 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-03-04T16:02:29.606 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-03-04T16:02:29.606 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-03-04T16:02:29.607 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-04T16:02:29.607 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-03-04T16:02:29.607 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-03-04T16:02:29.607 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-04T16:02:29.607 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-03-04T16:02:29.607 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-03-04T16:02:29.607 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-03-04T16:02:29.607 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-03-04T16:02:29.607 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-03-04T16:02:29.607 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-03-04T16:02:29.607 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-03-04T16:02:29.607 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-03-04T16:02:29.608 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-03-04T16:02:29.608 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-03-04T16:02:29.608 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-03-04T16:02:29.608 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-03-04T16:02:29.608 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-03-04T16:02:29.608 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-03-04T16:02:29.608 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-03-04T16:02:29.608 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-03-04T16:02:29.608 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-03-04T16:02:29.608 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-03-04T16:02:29.608 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-03-04T16:02:29.608 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-03-04T16:02:29.608 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-03-04T16:02:29.609 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-04T16:02:29.609 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-04T16:02:29.609 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-04T16:02:29.609 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-03-04T16:02:29.609 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-04T16:02:29.609 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-03-04T16:02:29.609 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-03-04T16:02:29.609 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-03-04T16:02:29.609 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-03-04T16:02:29.609 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-03-04T16:02:29.609 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-03-04T16:02:29.609 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-03-04T16:02:29.609 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-03-04T16:02:29.609 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-03-04T16:02:29.610 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-03-04T16:02:29.610 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-03-04T16:02:29.610 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-03-04T16:02:29.610 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-03-04T16:02:29.610 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-04T16:02:29.610 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-04T16:02:29.610 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-04T16:02:29.610 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-04T16:02:29.610 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-03-04T16:02:29.610 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-03-04T16:02:29.610 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-03-04T16:02:29.610 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-03-04T16:02:29.610 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-04T16:02:29.610 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-03-04T16:02:29.611 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-03-04T16:02:29.611 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-03-04T16:02:29.611 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-03-04T16:02:29.611 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-03-04T16:02:29.611 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-03-04T16:02:29.611 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-03-04T16:02:29.611 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-03-04T16:02:29.611 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-03-04T16:02:29.611 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-03-04T16:02:29.611 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-03-04T16:02:29.611 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-03-04T16:02:29.611 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-03-04T16:02:29.611 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-03-04T16:02:29.611 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-03-04T16:02:29.612 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-04T16:02:29.612 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-04T16:02:29.612 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-03-04T16:02:29.612 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-03-04T16:02:29.612 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-03-04T16:02:29.612 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-03-04T16:02:29.612 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-03-04T16:02:29.612 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-03-04T16:02:29.612 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-03-04T16:02:29.612 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-03-04T16:02:29.612 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-03-04T16:02:29.612 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-03-04T16:02:29.612 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-03-04T16:02:29.613 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-04T16:02:29.613 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-03-04T16:02:29.613 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-03-04T16:02:29.613 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-03-04T16:02:29.613 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-03-04T16:02:29.613 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-03-04T16:02:29.613 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-04T16:02:29.613 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-03-04T16:02:29.614 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-03-04T16:02:29.614 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-03-04T16:02:29.614 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-03-04T16:02:29.614 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-03-04T16:02:29.614 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-03-04T16:02:29.614 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-03-04T16:02:29.614 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-03-04T16:02:29.614 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-03-04T16:02:29.614 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-03-04T16:02:29.614 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-03-04T16:02:29.615 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-03-04T16:02:29.615 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-03-04T16:02:29.615 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-03-04T16:02:29.615 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-03-04T16:02:29.615 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-03-04T16:02:29.615 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-03-04T16:02:29.615 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-03-04T16:02:29.615 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-03-04T16:02:29.615 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-03-04T16:02:29.615 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-03-04T16:02:29.615 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-03-04T16:02:29.615 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-03-04T16:02:29.615 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-03-04T16:02:29.615 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-03-04T16:02:29.616 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-03-04T16:02:29.616 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-03-04T16:02:29.616 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-03-04T16:02:29.616 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-03-04T16:02:29.616 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-03-04T16:02:29.616 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-04T16:02:29.616 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-04T16:02:29.616 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-04T16:02:29.616 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-03-04T16:02:29.616 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-03-04T16:02:29.616 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-04T16:02:29.616 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-04T16:02:29.616 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-03-04T16:02:29.617 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-04T16:02:29.617 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-03-04T16:02:29.617 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-03-04T16:02:29.617 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-03-04T16:02:29.617 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-04T16:02:29.617 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-03-04T16:02:29.617 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-03-04T16:02:29.617 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-03-04T16:02:29.617 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-03-04T16:02:29.617 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-04T16:02:29.617 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-03-04T16:02:29.617 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-03-04T16:02:29.617 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-03-04T16:02:29.617 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-03-04T16:02:29.618 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-03-04T16:02:29.618 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-04T16:02:29.618 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-03-04T16:02:29.618 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-03-04T16:02:29.618 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-04T16:02:29.618 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-03-04T16:02:29.618 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-04T16:02:29.618 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-04T16:02:29.618 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-04T16:02:29.618 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-03-04T16:02:29.618 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-03-04T16:02:29.618 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-03-04T16:02:29.618 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-03-04T16:02:29.618 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-03-04T16:02:29.618 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-04T16:02:29.619 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-03-04T16:02:29.619 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-03-04T16:02:29.619 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-03-04T16:02:29.619 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-03-04T16:02:29.619 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-03-04T16:02:29.619 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-03-04T16:02:29.619 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-03-04T16:02:29.619 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-03-04T16:02:29.619 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-03-04T16:02:29.966 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-03-04T16:02:29.966 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-03-04T16:02:29.966 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-03-04T16:02:29.966 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-03-04T16:02:29.966 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-03-04T16:02:29.966 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-03-04T16:02:29.966 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-03-04T16:02:29.966 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-03-04T16:02:29.966 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-03-04T16:02:29.967 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-03-04T16:02:29.967 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-03-04T16:02:29.967 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-03-04T16:02:29.967 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-03-04T16:02:29.967 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-03-04T16:02:29.967 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-03-04T16:02:29.967 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-03-04T16:02:29.967 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-03-04T16:02:29.967 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-03-04T16:02:29.967 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-03-04T16:02:29.967 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-03-04T16:02:29.967 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-03-04T16:02:29.967 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-03-04T16:02:29.968 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-03-04T16:02:29.968 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-03-04T16:02:29.968 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-03-04T16:02:29.968 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-03-04T16:02:29.968 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-03-04T16:02:29.968 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-03-04T16:02:29.968 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-03-04T16:02:29.968 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-03-04T16:02:29.968 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-03-04T16:02:29.968 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-03-04T16:02:29.968 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-03-04T16:02:29.968 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-03-04T16:02:29.968 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=540.325485ms 2024-03-04T16:02:29.969 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-03-04T16:02:29.969 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-03-04T16:02:29.969 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-03-04T16:02:29.969 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-03-04T16:02:29.969 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-03-04T16:02:29.969 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-03-04T16:02:29.969 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-03-04T16:02:29.969 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-04T16:02:29.969 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-03-04T16:02:29.969 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-03-04T16:02:29.969 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:02:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:02:29+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-03-04T16:02:30.142 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-03-04T16:02:30.421 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:30 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:30.421 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:30 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:30.421 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:30 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:30.421 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:30 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:30.421 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:30 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:30.421 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:30 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:30.700 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:30 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:30.700 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:30 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:30.700 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:30 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:31.528 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:31 smithi086 ceph-mon[25584]: pgmap v50: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-04T16:02:31.721 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:31 smithi083 ceph-mon[24738]: pgmap v50: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-04T16:02:31.722 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:31 smithi083 ceph-mon[29452]: pgmap v50: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-04T16:02:32.357 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:02:32.357 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "e97e1ec0-7f35-4127-9c98-058f4e6e1ba6", 2024-03-04T16:02:32.357 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "z", 2024-03-04T16:02:32.357 INFO:teuthology.orchestra.run.smithi083.stdout: "domain_root": "z.rgw.meta:root", 2024-03-04T16:02:32.357 INFO:teuthology.orchestra.run.smithi083.stdout: "control_pool": "z.rgw.control", 2024-03-04T16:02:32.357 INFO:teuthology.orchestra.run.smithi083.stdout: "gc_pool": "z.rgw.log:gc", 2024-03-04T16:02:32.357 INFO:teuthology.orchestra.run.smithi083.stdout: "lc_pool": "z.rgw.log:lc", 2024-03-04T16:02:32.357 INFO:teuthology.orchestra.run.smithi083.stdout: "log_pool": "z.rgw.log", 2024-03-04T16:02:32.357 INFO:teuthology.orchestra.run.smithi083.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-03-04T16:02:32.357 INFO:teuthology.orchestra.run.smithi083.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-03-04T16:02:32.357 INFO:teuthology.orchestra.run.smithi083.stdout: "roles_pool": "z.rgw.meta:roles", 2024-03-04T16:02:32.358 INFO:teuthology.orchestra.run.smithi083.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-03-04T16:02:32.358 INFO:teuthology.orchestra.run.smithi083.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-03-04T16:02:32.358 INFO:teuthology.orchestra.run.smithi083.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-03-04T16:02:32.358 INFO:teuthology.orchestra.run.smithi083.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-03-04T16:02:32.358 INFO:teuthology.orchestra.run.smithi083.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-03-04T16:02:32.358 INFO:teuthology.orchestra.run.smithi083.stdout: "otp_pool": "z.rgw.otp", 2024-03-04T16:02:32.358 INFO:teuthology.orchestra.run.smithi083.stdout: "system_key": { 2024-03-04T16:02:32.358 INFO:teuthology.orchestra.run.smithi083.stdout: "access_key": "", 2024-03-04T16:02:32.358 INFO:teuthology.orchestra.run.smithi083.stdout: "secret_key": "" 2024-03-04T16:02:32.358 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:02:32.358 INFO:teuthology.orchestra.run.smithi083.stdout: "placement_pools": [ 2024-03-04T16:02:32.358 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-04T16:02:32.358 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "default-placement", 2024-03-04T16:02:32.358 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2024-03-04T16:02:32.359 INFO:teuthology.orchestra.run.smithi083.stdout: "index_pool": "z.rgw.buckets.index", 2024-03-04T16:02:32.359 INFO:teuthology.orchestra.run.smithi083.stdout: "storage_classes": { 2024-03-04T16:02:32.359 INFO:teuthology.orchestra.run.smithi083.stdout: "STANDARD": { 2024-03-04T16:02:32.359 INFO:teuthology.orchestra.run.smithi083.stdout: "data_pool": "z.rgw.buckets.data" 2024-03-04T16:02:32.359 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:02:32.359 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:02:32.359 INFO:teuthology.orchestra.run.smithi083.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-03-04T16:02:32.359 INFO:teuthology.orchestra.run.smithi083.stdout: "index_type": 0 2024-03-04T16:02:32.359 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:02:32.359 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:02:32.359 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-04T16:02:32.359 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_id": "dbdacf3e-1b44-4085-8338-e694476a974b", 2024-03-04T16:02:32.359 INFO:teuthology.orchestra.run.smithi083.stdout: "notif_pool": "z.rgw.log:notif" 2024-03-04T16:02:32.360 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:02:33.152 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-03-04T16:02:33.698 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:33 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:33.698 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:33 smithi083 ceph-mon[29452]: pgmap v51: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-04T16:02:33.698 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:33 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:33.699 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:33 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:33.699 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:33 smithi083 ceph-mon[24738]: pgmap v51: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-04T16:02:33.699 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:33 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:33.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:33 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:33.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:33 smithi086 ceph-mon[25584]: pgmap v51: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-04T16:02:33.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:33 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:36.113 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:02:35 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:02:35] "GET /metrics HTTP/1.1" 200 192805 "" "Prometheus/2.33.4" 2024-03-04T16:02:36.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:35 smithi083 ceph-mon[24738]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-04T16:02:36.114 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:35 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:36.114 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:35 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:36.114 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:35 smithi083 ceph-mon[24738]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-04T16:02:36.114 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:35 smithi083 ceph-mon[24738]: Reconfiguring daemon alertmanager.a on smithi083 2024-03-04T16:02:36.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:35 smithi083 ceph-mon[29452]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-04T16:02:36.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:35 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:36.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:35 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:36.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:35 smithi083 ceph-mon[29452]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-04T16:02:36.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:35 smithi083 ceph-mon[29452]: Reconfiguring daemon alertmanager.a on smithi083 2024-03-04T16:02:36.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:35 smithi086 ceph-mon[25584]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-04T16:02:36.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:35 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:36.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:35 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:36.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:35 smithi086 ceph-mon[25584]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-04T16:02:36.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:35 smithi086 ceph-mon[25584]: Reconfiguring daemon alertmanager.a on smithi083 2024-03-04T16:02:37.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:02:37 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:02:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:02:38.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:37 smithi086 ceph-mon[25584]: pgmap v53: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-04T16:02:38.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:37 smithi083 ceph-mon[24738]: pgmap v53: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-04T16:02:38.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:37 smithi083 ceph-mon[29452]: pgmap v53: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-04T16:02:39.178 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:38 smithi083 ceph-mon[24738]: osdmap e64: 8 total, 8 up, 8 in 2024-03-04T16:02:39.178 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:38 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1518303092' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-04T16:02:39.178 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:38 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:39.179 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:38 smithi083 ceph-mon[29452]: osdmap e64: 8 total, 8 up, 8 in 2024-03-04T16:02:39.179 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:38 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1518303092' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-04T16:02:39.179 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:38 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:39.179 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:38 smithi083 systemd[1]: Stopping Ceph alertmanager.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:02:39.179 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:38 smithi083 bash[49390]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager.a" found: no such container 2024-03-04T16:02:39.179 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:38 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[46972]: level=info ts=2024-03-04T16:02:38.935Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-04T16:02:39.179 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:38 smithi083 podman[49402]: 2024-03-04 16:02:38.955937337 +0000 UTC m=+0.126076035 container died e9d134cf60fb32a254019f7cbf42b1d944eb3945fab9d8bf7a0f5eb1ac26d8cc (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:02:39.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:38 smithi086 ceph-mon[25584]: osdmap e64: 8 total, 8 up, 8 in 2024-03-04T16:02:39.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:38 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1518303092' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-04T16:02:39.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:38 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:39.461 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:39 smithi083 podman[49402]: 2024-03-04 16:02:39.268996881 +0000 UTC m=+0.439135586 container cleanup e9d134cf60fb32a254019f7cbf42b1d944eb3945fab9d8bf7a0f5eb1ac26d8cc (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:02:39.462 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:39 smithi083 bash[49402]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a 2024-03-04T16:02:39.739 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:39 smithi083 podman[49417]: 2024-03-04 16:02:39.46168096 +0000 UTC m=+0.505882958 container remove e9d134cf60fb32a254019f7cbf42b1d944eb3945fab9d8bf7a0f5eb1ac26d8cc (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:02:39.739 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:39 smithi083 podman[49417]: 2024-03-04 16:02:39.545678704 +0000 UTC m=+0.589880713 volume remove 317fab3c0708e7f8a9ff5586175b0099a3571a43da86a0f0ec7108fd5e727605 2024-03-04T16:02:39.739 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:39 smithi083 bash[49431]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager.a" found: no such container 2024-03-04T16:02:39.739 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:39 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-04T16:02:39.739 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:39 smithi083 systemd[1]: Stopped Ceph alertmanager.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:02:39.740 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:39 smithi083 systemd[1]: Starting Ceph alertmanager.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:02:40.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:39 smithi083 ceph-mon[29452]: pgmap v55: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-04T16:02:40.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:39 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1518303092' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-04T16:02:40.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:39 smithi083 ceph-mon[29452]: osdmap e65: 8 total, 8 up, 8 in 2024-03-04T16:02:40.179 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:39 smithi083 ceph-mon[24738]: pgmap v55: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-04T16:02:40.179 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:39 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1518303092' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-04T16:02:40.179 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:39 smithi083 ceph-mon[24738]: osdmap e65: 8 total, 8 up, 8 in 2024-03-04T16:02:40.179 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:39 smithi083 podman[49500]: 2024-03-04 16:02:39.758776037 +0000 UTC m=+0.017593685 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-04T16:02:40.180 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:39 smithi083 podman[49500]: 2024-03-04 16:02:39.871550661 +0000 UTC m=+0.130368352 volume create 2ac12bd63644fc0ccd25c86394196b2422cce79643cf06b737b8c58e696f3333 2024-03-04T16:02:40.180 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:39 smithi083 podman[49500]: 2024-03-04 16:02:39.980560934 +0000 UTC m=+0.239378584 container create 8a58c6dd3c10921b2be122535071431d8504a354f17b43abb36923dbd191dde3 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:02:40.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:39 smithi086 ceph-mon[25584]: pgmap v55: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-04T16:02:40.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:39 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1518303092' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-04T16:02:40.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:39 smithi086 ceph-mon[25584]: osdmap e65: 8 total, 8 up, 8 in 2024-03-04T16:02:40.430 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:40 smithi083 podman[49500]: 2024-03-04 16:02:40.308697108 +0000 UTC m=+0.567514767 container init 8a58c6dd3c10921b2be122535071431d8504a354f17b43abb36923dbd191dde3 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:02:40.430 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:40 smithi083 podman[49500]: 2024-03-04 16:02:40.311295495 +0000 UTC m=+0.570113148 container start 8a58c6dd3c10921b2be122535071431d8504a354f17b43abb36923dbd191dde3 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:02:40.431 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:40 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=info ts=2024-03-04T16:02:40.321Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-04T16:02:40.431 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:40 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=info ts=2024-03-04T16:02:40.321Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-04T16:02:40.431 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:40 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=info ts=2024-03-04T16:02:40.322Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.83 port=9094 2024-03-04T16:02:40.431 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:40 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=info ts=2024-03-04T16:02:40.323Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-04T16:02:40.431 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:40 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=info ts=2024-03-04T16:02:40.348Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-04T16:02:40.431 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:40 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=info ts=2024-03-04T16:02:40.348Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-04T16:02:40.431 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:40 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=info ts=2024-03-04T16:02:40.350Z caller=main.go:518 msg=Listening address=:9093 2024-03-04T16:02:40.431 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:40 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=info ts=2024-03-04T16:02:40.350Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-04T16:02:40.431 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:40 smithi083 bash[49500]: 8a58c6dd3c10921b2be122535071431d8504a354f17b43abb36923dbd191dde3 2024-03-04T16:02:40.431 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:40 smithi083 systemd[1]: Started Ceph alertmanager.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:02:41.107 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:40 smithi086 ceph-mon[25584]: osdmap e66: 8 total, 8 up, 8 in 2024-03-04T16:02:41.107 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:40 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1518303092' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-04T16:02:41.107 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:40 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:41.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:40 smithi083 ceph-mon[24738]: osdmap e66: 8 total, 8 up, 8 in 2024-03-04T16:02:41.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:40 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1518303092' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-04T16:02:41.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:40 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:41.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:40 smithi083 ceph-mon[29452]: osdmap e66: 8 total, 8 up, 8 in 2024-03-04T16:02:41.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:40 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1518303092' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-04T16:02:41.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:40 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:41.918 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:41 smithi086 ceph-mon[25584]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-04T16:02:41.918 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:41 smithi086 ceph-mon[25584]: Reconfiguring daemon prometheus.a on smithi086 2024-03-04T16:02:42.225 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:41 smithi086 ceph-mon[25584]: pgmap v58: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 3 op/s 2024-03-04T16:02:42.225 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:41 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1518303092' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-04T16:02:42.225 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:41 smithi086 ceph-mon[25584]: osdmap e67: 8 total, 8 up, 8 in 2024-03-04T16:02:42.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:41 smithi083 ceph-mon[29452]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-04T16:02:42.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:41 smithi083 ceph-mon[29452]: Reconfiguring daemon prometheus.a on smithi086 2024-03-04T16:02:42.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:41 smithi083 ceph-mon[29452]: pgmap v58: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 3 op/s 2024-03-04T16:02:42.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:41 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1518303092' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-04T16:02:42.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:41 smithi083 ceph-mon[29452]: osdmap e67: 8 total, 8 up, 8 in 2024-03-04T16:02:42.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:41 smithi083 ceph-mon[24738]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-04T16:02:42.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:41 smithi083 ceph-mon[24738]: Reconfiguring daemon prometheus.a on smithi086 2024-03-04T16:02:42.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:41 smithi083 ceph-mon[24738]: pgmap v58: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 3 op/s 2024-03-04T16:02:42.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:41 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1518303092' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-04T16:02:42.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:41 smithi083 ceph-mon[24738]: osdmap e67: 8 total, 8 up, 8 in 2024-03-04T16:02:42.579 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:42 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=info ts=2024-03-04T16:02:42.324Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000416008s 2024-03-04T16:02:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:42 smithi086 ceph-mon[25584]: osdmap e68: 8 total, 8 up, 8 in 2024-03-04T16:02:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:42 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3079817626' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T16:02:43.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:42 smithi086 ceph-mon[25584]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T16:02:43.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:42 smithi083 ceph-mon[24738]: osdmap e68: 8 total, 8 up, 8 in 2024-03-04T16:02:43.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:42 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3079817626' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T16:02:43.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:42 smithi083 ceph-mon[24738]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T16:02:43.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:42 smithi083 ceph-mon[29452]: osdmap e68: 8 total, 8 up, 8 in 2024-03-04T16:02:43.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:42 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3079817626' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T16:02:43.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:42 smithi083 ceph-mon[29452]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T16:02:43.787 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:43 smithi086 systemd[1]: Stopping Ceph prometheus.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:02:43.788 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:43 smithi086 bash[42284]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus.a" found: no such container 2024-03-04T16:02:44.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-mon[25584]: pgmap v61: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-03-04T16:02:44.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-mon[25584]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-04T16:02:44.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-mon[25584]: osdmap e69: 8 total, 8 up, 8 in 2024-03-04T16:02:44.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3079817626' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T16:02:44.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-mon[25584]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T16:02:44.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-mon[25584]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T16:02:44.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-mon[25584]: osdmap e70: 8 total, 8 up, 8 in 2024-03-04T16:02:44.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3079817626' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-04T16:02:44.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-mon[25584]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-04T16:02:44.089 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:02:43.787Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-04T16:02:44.089 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:02:43.787Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-04T16:02:44.089 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:02:43.787Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-04T16:02:44.089 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:02:43.787Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-04T16:02:44.089 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:02:43.787Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-04T16:02:44.089 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:02:43.787Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-04T16:02:44.089 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:02:43.787Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-04T16:02:44.089 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:02:43.787Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-04T16:02:44.089 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:02:43.787Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-04T16:02:44.089 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:02:43.788Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-04T16:02:44.089 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:02:43.788Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-04T16:02:44.089 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:43 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[40615]: ts=2024-03-04T16:02:43.788Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-04T16:02:44.089 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:43 smithi086 podman[42297]: 2024-03-04 16:02:43.797164783 +0000 UTC m=+0.111078311 container died 54f5ce4438a69e5889c06310872a3d0f292592714800309e39b3f69a01b7c614 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:02:44.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:43 smithi083 ceph-mon[29452]: pgmap v61: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-03-04T16:02:44.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:43 smithi083 ceph-mon[29452]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-04T16:02:44.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:43 smithi083 ceph-mon[29452]: osdmap e69: 8 total, 8 up, 8 in 2024-03-04T16:02:44.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:43 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3079817626' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T16:02:44.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:43 smithi083 ceph-mon[29452]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T16:02:44.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:43 smithi083 ceph-mon[29452]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T16:02:44.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:43 smithi083 ceph-mon[29452]: osdmap e70: 8 total, 8 up, 8 in 2024-03-04T16:02:44.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:43 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3079817626' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-04T16:02:44.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:43 smithi083 ceph-mon[29452]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-04T16:02:44.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:43 smithi083 ceph-mon[24738]: pgmap v61: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-03-04T16:02:44.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:43 smithi083 ceph-mon[24738]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-04T16:02:44.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:43 smithi083 ceph-mon[24738]: osdmap e69: 8 total, 8 up, 8 in 2024-03-04T16:02:44.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:43 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3079817626' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T16:02:44.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:43 smithi083 ceph-mon[24738]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T16:02:44.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:43 smithi083 ceph-mon[24738]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T16:02:44.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:43 smithi083 ceph-mon[24738]: osdmap e70: 8 total, 8 up, 8 in 2024-03-04T16:02:44.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:43 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3079817626' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-04T16:02:44.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:43 smithi083 ceph-mon[24738]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-04T16:02:44.454 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 podman[42297]: 2024-03-04 16:02:44.178235049 +0000 UTC m=+0.492148578 container cleanup 54f5ce4438a69e5889c06310872a3d0f292592714800309e39b3f69a01b7c614 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:02:44.455 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 bash[42297]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a 2024-03-04T16:02:44.455 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 bash[42324]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus.a" found: no such container 2024-03-04T16:02:44.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 podman[42311]: 2024-03-04 16:02:44.454586265 +0000 UTC m=+0.659447949 container remove 54f5ce4438a69e5889c06310872a3d0f292592714800309e39b3f69a01b7c614 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:02:44.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-04T16:02:44.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 systemd[1]: Stopped Ceph prometheus.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:02:44.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 systemd[1]: Starting Ceph prometheus.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:02:44.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 podman[42391]: 2024-03-04 16:02:44.593700047 +0000 UTC m=+0.018848406 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-04T16:02:44.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 podman[42391]: 2024-03-04 16:02:44.697311131 +0000 UTC m=+0.122459533 container create 82e50b4a8b864d533e268dba33b9e5487364e8c28580a310eeca80033a6f5019 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:02:44.981 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:02:44.981 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "00c5ce8a-2f43-4c07-a2ea-fce44aa94452", 2024-03-04T16:02:44.981 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2024-03-04T16:02:44.981 INFO:teuthology.orchestra.run.smithi083.stdout: "predecessor_uuid": "268b3a83-baa8-4f22-badb-7330b2a001b7", 2024-03-04T16:02:44.981 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": [], 2024-03-04T16:02:44.981 INFO:teuthology.orchestra.run.smithi083.stdout: "period_map": { 2024-03-04T16:02:44.981 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "00c5ce8a-2f43-4c07-a2ea-fce44aa94452", 2024-03-04T16:02:44.981 INFO:teuthology.orchestra.run.smithi083.stdout: "zonegroups": [ 2024-03-04T16:02:44.981 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-04T16:02:44.981 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "c0f800a7-05b5-44df-ba36-43c7bf5fbcbb", 2024-03-04T16:02:44.981 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "default", 2024-03-04T16:02:44.982 INFO:teuthology.orchestra.run.smithi083.stdout: "api_name": "default", 2024-03-04T16:02:44.982 INFO:teuthology.orchestra.run.smithi083.stdout: "is_master": "true", 2024-03-04T16:02:44.982 INFO:teuthology.orchestra.run.smithi083.stdout: "endpoints": [], 2024-03-04T16:02:44.982 INFO:teuthology.orchestra.run.smithi083.stdout: "hostnames": [], 2024-03-04T16:02:44.982 INFO:teuthology.orchestra.run.smithi083.stdout: "hostnames_s3website": [], 2024-03-04T16:02:44.982 INFO:teuthology.orchestra.run.smithi083.stdout: "master_zone": "e97e1ec0-7f35-4127-9c98-058f4e6e1ba6", 2024-03-04T16:02:44.982 INFO:teuthology.orchestra.run.smithi083.stdout: "zones": [ 2024-03-04T16:02:44.982 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-04T16:02:44.982 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "e97e1ec0-7f35-4127-9c98-058f4e6e1ba6", 2024-03-04T16:02:44.982 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "z", 2024-03-04T16:02:44.982 INFO:teuthology.orchestra.run.smithi083.stdout: "endpoints": [], 2024-03-04T16:02:44.982 INFO:teuthology.orchestra.run.smithi083.stdout: "log_meta": "false", 2024-03-04T16:02:44.982 INFO:teuthology.orchestra.run.smithi083.stdout: "log_data": "false", 2024-03-04T16:02:44.982 INFO:teuthology.orchestra.run.smithi083.stdout: "bucket_index_max_shards": 11, 2024-03-04T16:02:44.982 INFO:teuthology.orchestra.run.smithi083.stdout: "read_only": "false", 2024-03-04T16:02:44.983 INFO:teuthology.orchestra.run.smithi083.stdout: "tier_type": "", 2024-03-04T16:02:44.983 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_from_all": "true", 2024-03-04T16:02:44.983 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_from": [], 2024-03-04T16:02:44.983 INFO:teuthology.orchestra.run.smithi083.stdout: "redirect_zone": "" 2024-03-04T16:02:44.983 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:02:44.983 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-04T16:02:44.983 INFO:teuthology.orchestra.run.smithi083.stdout: "placement_targets": [ 2024-03-04T16:02:44.983 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-04T16:02:44.983 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "default-placement", 2024-03-04T16:02:44.983 INFO:teuthology.orchestra.run.smithi083.stdout: "tags": [], 2024-03-04T16:02:44.983 INFO:teuthology.orchestra.run.smithi083.stdout: "storage_classes": [ 2024-03-04T16:02:44.983 INFO:teuthology.orchestra.run.smithi083.stdout: "STANDARD" 2024-03-04T16:02:44.984 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-03-04T16:02:44.984 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:02:44.984 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-04T16:02:44.984 INFO:teuthology.orchestra.run.smithi083.stdout: "default_placement": "default-placement", 2024-03-04T16:02:44.984 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_id": "dbdacf3e-1b44-4085-8338-e694476a974b", 2024-03-04T16:02:44.984 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_policy": { 2024-03-04T16:02:44.984 INFO:teuthology.orchestra.run.smithi083.stdout: "groups": [] 2024-03-04T16:02:44.984 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:02:44.984 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:02:44.984 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-04T16:02:44.984 INFO:teuthology.orchestra.run.smithi083.stdout: "short_zone_ids": [ 2024-03-04T16:02:44.984 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-04T16:02:44.984 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "e97e1ec0-7f35-4127-9c98-058f4e6e1ba6", 2024-03-04T16:02:44.984 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 2668000650 2024-03-04T16:02:44.985 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:02:44.985 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-03-04T16:02:44.985 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:02:44.985 INFO:teuthology.orchestra.run.smithi083.stdout: "master_zonegroup": "c0f800a7-05b5-44df-ba36-43c7bf5fbcbb", 2024-03-04T16:02:44.985 INFO:teuthology.orchestra.run.smithi083.stdout: "master_zone": "e97e1ec0-7f35-4127-9c98-058f4e6e1ba6", 2024-03-04T16:02:44.985 INFO:teuthology.orchestra.run.smithi083.stdout: "period_config": { 2024-03-04T16:02:44.985 INFO:teuthology.orchestra.run.smithi083.stdout: "bucket_quota": { 2024-03-04T16:02:44.985 INFO:teuthology.orchestra.run.smithi083.stdout: "enabled": false, 2024-03-04T16:02:44.985 INFO:teuthology.orchestra.run.smithi083.stdout: "check_on_raw": false, 2024-03-04T16:02:44.985 INFO:teuthology.orchestra.run.smithi083.stdout: "max_size": -1, 2024-03-04T16:02:44.985 INFO:teuthology.orchestra.run.smithi083.stdout: "max_size_kb": 0, 2024-03-04T16:02:44.985 INFO:teuthology.orchestra.run.smithi083.stdout: "max_objects": -1 2024-03-04T16:02:44.985 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:02:44.985 INFO:teuthology.orchestra.run.smithi083.stdout: "user_quota": { 2024-03-04T16:02:44.985 INFO:teuthology.orchestra.run.smithi083.stdout: "enabled": false, 2024-03-04T16:02:44.986 INFO:teuthology.orchestra.run.smithi083.stdout: "check_on_raw": false, 2024-03-04T16:02:44.986 INFO:teuthology.orchestra.run.smithi083.stdout: "max_size": -1, 2024-03-04T16:02:44.986 INFO:teuthology.orchestra.run.smithi083.stdout: "max_size_kb": 0, 2024-03-04T16:02:44.986 INFO:teuthology.orchestra.run.smithi083.stdout: "max_objects": -1 2024-03-04T16:02:44.986 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:02:44.986 INFO:teuthology.orchestra.run.smithi083.stdout: "user_ratelimit": { 2024-03-04T16:02:44.986 INFO:teuthology.orchestra.run.smithi083.stdout: "max_read_ops": 0, 2024-03-04T16:02:44.986 INFO:teuthology.orchestra.run.smithi083.stdout: "max_write_ops": 0, 2024-03-04T16:02:44.986 INFO:teuthology.orchestra.run.smithi083.stdout: "max_read_bytes": 0, 2024-03-04T16:02:44.986 INFO:teuthology.orchestra.run.smithi083.stdout: "max_write_bytes": 0, 2024-03-04T16:02:44.986 INFO:teuthology.orchestra.run.smithi083.stdout: "enabled": false 2024-03-04T16:02:44.986 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:02:44.986 INFO:teuthology.orchestra.run.smithi083.stdout: "bucket_ratelimit": { 2024-03-04T16:02:44.986 INFO:teuthology.orchestra.run.smithi083.stdout: "max_read_ops": 0, 2024-03-04T16:02:44.986 INFO:teuthology.orchestra.run.smithi083.stdout: "max_write_ops": 0, 2024-03-04T16:02:44.987 INFO:teuthology.orchestra.run.smithi083.stdout: "max_read_bytes": 0, 2024-03-04T16:02:44.987 INFO:teuthology.orchestra.run.smithi083.stdout: "max_write_bytes": 0, 2024-03-04T16:02:44.987 INFO:teuthology.orchestra.run.smithi083.stdout: "enabled": false 2024-03-04T16:02:44.987 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:02:44.987 INFO:teuthology.orchestra.run.smithi083.stdout: "anonymous_ratelimit": { 2024-03-04T16:02:44.987 INFO:teuthology.orchestra.run.smithi083.stdout: "max_read_ops": 0, 2024-03-04T16:02:44.987 INFO:teuthology.orchestra.run.smithi083.stdout: "max_write_ops": 0, 2024-03-04T16:02:44.987 INFO:teuthology.orchestra.run.smithi083.stdout: "max_read_bytes": 0, 2024-03-04T16:02:44.987 INFO:teuthology.orchestra.run.smithi083.stdout: "max_write_bytes": 0, 2024-03-04T16:02:44.987 INFO:teuthology.orchestra.run.smithi083.stdout: "enabled": false 2024-03-04T16:02:44.987 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:02:44.987 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:02:44.987 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_id": "dbdacf3e-1b44-4085-8338-e694476a974b", 2024-03-04T16:02:44.987 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_name": "r", 2024-03-04T16:02:44.988 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2024-03-04T16:02:44.988 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:02:45.131 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 podman[42391]: 2024-03-04 16:02:44.916078406 +0000 UTC m=+0.341226777 container init 82e50b4a8b864d533e268dba33b9e5487364e8c28580a310eeca80033a6f5019 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:02:45.132 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 podman[42391]: 2024-03-04 16:02:44.9186941 +0000 UTC m=+0.343842463 container start 82e50b4a8b864d533e268dba33b9e5487364e8c28580a310eeca80033a6f5019 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:02:45.132 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:44.945Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-04T16:02:45.132 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:44.945Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-04T16:02:45.132 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:44.945Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-04T16:02:45.132 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:44.945Z caller=main.go:518 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi086 (none))" 2024-03-04T16:02:45.132 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:44.945Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-04T16:02:45.132 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:44.945Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-04T16:02:45.132 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:44.946Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-04T16:02:45.132 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:44.947Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-04T16:02:45.132 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:44.948Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-04T16:02:45.132 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:44.950Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-04T16:02:45.132 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:44.950Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.636µs 2024-03-04T16:02:45.132 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:44.950Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-04T16:02:45.133 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:45 smithi086 bash[42391]: 82e50b4a8b864d533e268dba33b9e5487364e8c28580a310eeca80033a6f5019 2024-03-04T16:02:45.133 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:45 smithi086 systemd[1]: Started Ceph prometheus.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:02:45.484 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:45.157Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-04T16:02:45.484 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:45.158Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-04T16:02:45.484 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:45.158Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=23.961µs wal_replay_duration=207.207483ms total_replay_duration=207.245056ms 2024-03-04T16:02:45.484 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:45.159Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-04T16:02:45.485 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:45.160Z caller=main.go:947 level=info msg="TSDB started" 2024-03-04T16:02:45.485 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:45.160Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-04T16:02:45.485 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:45.176Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.691543ms db_storage=685ns remote_storage=1.182µs web_handler=371ns query_engine=697ns scrape=837.055µs scrape_sd=45.958µs notify=39.981µs notify_sd=12.674µs rules=15.40016ms 2024-03-04T16:02:45.485 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:02:45.176Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-04T16:02:45.644 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-03-04T16:02:46.186 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[24738]: pgmap v64: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-04T16:02:46.186 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[24738]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-04T16:02:46.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[24738]: osdmap e71: 8 total, 8 up, 8 in 2024-03-04T16:02:46.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:46.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:02:46.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.83:9093"}]: dispatch 2024-03-04T16:02:46.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:46.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:02:46.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.86:9095"}]: dispatch 2024-03-04T16:02:46.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:46.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:02:46.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.86:3000"}]: dispatch 2024-03-04T16:02:46.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:46.188 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:46.188 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:46.188 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[29452]: pgmap v64: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-04T16:02:46.188 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[29452]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-04T16:02:46.188 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[29452]: osdmap e71: 8 total, 8 up, 8 in 2024-03-04T16:02:46.188 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:46.188 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:02:46.189 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.83:9093"}]: dispatch 2024-03-04T16:02:46.189 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:46.189 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:02:46.189 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.86:9095"}]: dispatch 2024-03-04T16:02:46.189 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:46.189 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:02:46.189 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.86:3000"}]: dispatch 2024-03-04T16:02:46.189 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:46.189 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:46.189 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:45 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:46.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-mon[25584]: pgmap v64: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-04T16:02:46.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-mon[25584]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-04T16:02:46.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-mon[25584]: osdmap e71: 8 total, 8 up, 8 in 2024-03-04T16:02:46.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:46.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:02:46.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.83:9093"}]: dispatch 2024-03-04T16:02:46.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:46.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:02:46.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.86:9095"}]: dispatch 2024-03-04T16:02:46.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:46.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:02:46.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.86:3000"}]: dispatch 2024-03-04T16:02:46.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:46.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:46.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:45 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:47.219 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:46 smithi086 ceph-mon[25584]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:02:47.219 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:46 smithi086 ceph-mon[25584]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.83:9093"}]: dispatch 2024-03-04T16:02:47.219 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:46 smithi086 ceph-mon[25584]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:02:47.219 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:46 smithi086 ceph-mon[25584]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.86:9095"}]: dispatch 2024-03-04T16:02:47.220 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:46 smithi086 ceph-mon[25584]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:02:47.220 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:46 smithi086 ceph-mon[25584]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.86:3000"}]: dispatch 2024-03-04T16:02:47.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:46 smithi083 ceph-mon[24738]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:02:47.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:46 smithi083 ceph-mon[24738]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.83:9093"}]: dispatch 2024-03-04T16:02:47.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:46 smithi083 ceph-mon[24738]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:02:47.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:46 smithi083 ceph-mon[24738]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.86:9095"}]: dispatch 2024-03-04T16:02:47.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:46 smithi083 ceph-mon[24738]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:02:47.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:46 smithi083 ceph-mon[24738]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.86:3000"}]: dispatch 2024-03-04T16:02:47.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:46 smithi083 ceph-mon[29452]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:02:47.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:46 smithi083 ceph-mon[29452]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.83:9093"}]: dispatch 2024-03-04T16:02:47.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:46 smithi083 ceph-mon[29452]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:02:47.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:46 smithi083 ceph-mon[29452]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.86:9095"}]: dispatch 2024-03-04T16:02:47.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:46 smithi083 ceph-mon[29452]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:02:47.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:46 smithi083 ceph-mon[29452]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.86:3000"}]: dispatch 2024-03-04T16:02:47.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:47 smithi083 ceph-mon[24738]: pgmap v66: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 208 B/s rd, 417 B/s wr, 1 op/s 2024-03-04T16:02:47.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:47 smithi083 ceph-mon[29452]: pgmap v66: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 208 B/s rd, 417 B/s wr, 1 op/s 2024-03-04T16:02:48.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:47 smithi086 ceph-mon[25584]: pgmap v66: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 208 B/s rd, 417 B/s wr, 1 op/s 2024-03-04T16:02:48.497 INFO:teuthology.orchestra.run.smithi083.stdout:Scheduled rgw.foo update... 2024-03-04T16:02:49.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:49 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:49.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:49 smithi083 ceph-mon[24738]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:49.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:49 smithi083 ceph-mon[24738]: Saving service rgw.foo spec with placement count:2 2024-03-04T16:02:49.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:49 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:49.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:49 smithi083 ceph-mon[24738]: pgmap v67: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 511 B/s wr, 1 op/s 2024-03-04T16:02:49.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:49 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:49.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:49 smithi083 ceph-mon[29452]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:49.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:49 smithi083 ceph-mon[29452]: Saving service rgw.foo spec with placement count:2 2024-03-04T16:02:49.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:49 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:49.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:49 smithi083 ceph-mon[29452]: pgmap v67: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 511 B/s wr, 1 op/s 2024-03-04T16:02:49.526 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph orch apply rgw smpl' 2024-03-04T16:02:49.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:49 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:49.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:49 smithi086 ceph-mon[25584]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:49.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:49 smithi086 ceph-mon[25584]: Saving service rgw.foo spec with placement count:2 2024-03-04T16:02:49.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:49 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:49.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:49 smithi086 ceph-mon[25584]: pgmap v67: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 511 B/s wr, 1 op/s 2024-03-04T16:02:50.738 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:02:50 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=info ts=2024-03-04T16:02:50.326Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002906925s 2024-03-04T16:02:51.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:51 smithi083 ceph-mon[24738]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-04T16:02:51.337 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:51 smithi083 ceph-mon[29452]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-04T16:02:51.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:51 smithi086 ceph-mon[25584]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-04T16:02:52.414 INFO:teuthology.orchestra.run.smithi083.stdout:Scheduled rgw.smpl update... 2024-03-04T16:02:52.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:52.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:52.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:52.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:52.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[24738]: Saving service rgw.foo spec with placement count:2 2024-03-04T16:02:52.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:52.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi083.szjzkw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:02:52.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi083.szjzkw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:02:52.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi083.szjzkw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T16:02:52.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:52.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:52.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[24738]: Deploying daemon rgw.foo.smithi083.szjzkw on smithi083 2024-03-04T16:02:52.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:52.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:52.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:52.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:52.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:52.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[29452]: Saving service rgw.foo spec with placement count:2 2024-03-04T16:02:52.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:52.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi083.szjzkw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:02:52.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi083.szjzkw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:02:52.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi083.szjzkw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T16:02:52.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:52.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:52.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[29452]: Deploying daemon rgw.foo.smithi083.szjzkw on smithi083 2024-03-04T16:02:52.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:52 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:53.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:52 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:53.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:52 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:53.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:52 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:53.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:52 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:53.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:52 smithi086 ceph-mon[25584]: Saving service rgw.foo spec with placement count:2 2024-03-04T16:02:53.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:52 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:53.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:52 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi083.szjzkw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:02:53.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:52 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi083.szjzkw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:02:53.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:52 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi083.szjzkw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T16:02:53.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:52 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:53.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:52 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:53.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:52 smithi086 ceph-mon[25584]: Deploying daemon rgw.foo.smithi083.szjzkw on smithi083 2024-03-04T16:02:53.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:52 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:53.681 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph osd pool create foo' 2024-03-04T16:02:53.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:53 smithi083 ceph-mon[29452]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:53.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:53 smithi083 ceph-mon[29452]: Saving service rgw.smpl spec with placement count:2 2024-03-04T16:02:53.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:53 smithi083 ceph-mon[29452]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 805 B/s rd, 805 B/s wr, 2 op/s 2024-03-04T16:02:53.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:53 smithi083 ceph-mon[24738]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:53.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:53 smithi083 ceph-mon[24738]: Saving service rgw.smpl spec with placement count:2 2024-03-04T16:02:53.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:53 smithi083 ceph-mon[24738]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 805 B/s rd, 805 B/s wr, 2 op/s 2024-03-04T16:02:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:53 smithi086 ceph-mon[25584]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:53 smithi086 ceph-mon[25584]: Saving service rgw.smpl spec with placement count:2 2024-03-04T16:02:54.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:53 smithi086 ceph-mon[25584]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 805 B/s rd, 805 B/s wr, 2 op/s 2024-03-04T16:02:56.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:02:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:02:55] "GET /metrics HTTP/1.1" 200 196807 "" "Prometheus/2.33.4" 2024-03-04T16:02:56.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:55 smithi083 ceph-mon[29452]: pgmap v70: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-03-04T16:02:56.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:55 smithi083 ceph-mon[24738]: pgmap v70: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-03-04T16:02:56.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:55 smithi086 ceph-mon[25584]: pgmap v70: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-03-04T16:02:56.891 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'foo' created 2024-03-04T16:02:56.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:56 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1824057583' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-04T16:02:56.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:56 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:56.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:56 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.ezumfo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:02:56.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:56 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.ezumfo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:02:56.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:56 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.ezumfo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T16:02:56.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:56 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:56.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:56 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:57.214 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:56 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1824057583' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-04T16:02:57.214 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:56 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:57.214 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:56 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.ezumfo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:02:57.214 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:56 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.ezumfo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:02:57.214 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:56 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.ezumfo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T16:02:57.214 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:56 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:57.214 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:56 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:57.215 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:56 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1824057583' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-04T16:02:57.215 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:56 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:57.215 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:56 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.ezumfo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:02:57.215 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:56 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.ezumfo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:02:57.215 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:56 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.ezumfo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T16:02:57.215 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:56 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:02:57.215 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:56 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:57.653 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:02:57 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:02:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:02:57.671 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'rbd pool init foo' 2024-03-04T16:02:57.979 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:57 smithi083 ceph-mon[29452]: Deploying daemon rgw.foo.smithi086.ezumfo on smithi086 2024-03-04T16:02:57.979 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:57 smithi083 ceph-mon[29452]: pgmap v71: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 602 B/s wr, 1 op/s 2024-03-04T16:02:57.979 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:57 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1824057583' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-04T16:02:57.979 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:57 smithi083 ceph-mon[29452]: osdmap e72: 8 total, 8 up, 8 in 2024-03-04T16:02:57.980 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:57 smithi083 ceph-mon[24738]: Deploying daemon rgw.foo.smithi086.ezumfo on smithi086 2024-03-04T16:02:57.980 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:57 smithi083 ceph-mon[24738]: pgmap v71: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 602 B/s wr, 1 op/s 2024-03-04T16:02:57.980 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:57 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1824057583' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-04T16:02:57.980 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:57 smithi083 ceph-mon[24738]: osdmap e72: 8 total, 8 up, 8 in 2024-03-04T16:02:57.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:57 smithi086 ceph-mon[25584]: Deploying daemon rgw.foo.smithi086.ezumfo on smithi086 2024-03-04T16:02:57.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:57 smithi086 ceph-mon[25584]: pgmap v71: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 602 B/s wr, 1 op/s 2024-03-04T16:02:57.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:57 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1824057583' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-04T16:02:57.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:57 smithi086 ceph-mon[25584]: osdmap e72: 8 total, 8 up, 8 in 2024-03-04T16:02:58.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:58 smithi086 ceph-mon[25584]: osdmap e73: 8 total, 8 up, 8 in 2024-03-04T16:02:59.171 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:58 smithi083 ceph-mon[29452]: osdmap e73: 8 total, 8 up, 8 in 2024-03-04T16:02:59.172 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:58 smithi083 ceph-mon[24738]: osdmap e73: 8 total, 8 up, 8 in 2024-03-04T16:03:00.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:59 smithi086 ceph-mon[25584]: pgmap v74: 161 pgs: 15 unknown, 146 active+clean; 582 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 2.1 KiB/s wr, 67 op/s 2024-03-04T16:03:00.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:59 smithi086 ceph-mon[25584]: osdmap e74: 8 total, 8 up, 8 in 2024-03-04T16:03:00.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:59 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:00.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:59 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:00.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:59 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:00.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:59 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3836849224' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-04T16:03:00.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:02:59 smithi086 ceph-mon[25584]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-04T16:03:00.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:59 smithi083 ceph-mon[24738]: pgmap v74: 161 pgs: 15 unknown, 146 active+clean; 582 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 2.1 KiB/s wr, 67 op/s 2024-03-04T16:03:00.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:59 smithi083 ceph-mon[24738]: osdmap e74: 8 total, 8 up, 8 in 2024-03-04T16:03:00.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:59 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:00.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:59 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:00.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:59 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:00.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:59 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3836849224' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-04T16:03:00.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:02:59 smithi083 ceph-mon[24738]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-04T16:03:00.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:59 smithi083 ceph-mon[29452]: pgmap v74: 161 pgs: 15 unknown, 146 active+clean; 582 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 2.1 KiB/s wr, 67 op/s 2024-03-04T16:03:00.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:59 smithi083 ceph-mon[29452]: osdmap e74: 8 total, 8 up, 8 in 2024-03-04T16:03:00.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:59 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:00.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:59 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:00.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:59 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:00.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:59 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3836849224' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-04T16:03:00.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:02:59 smithi083 ceph-mon[29452]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-04T16:03:00.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:00 smithi083 ceph-mon[29452]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-04T16:03:00.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:00 smithi083 ceph-mon[29452]: osdmap e75: 8 total, 8 up, 8 in 2024-03-04T16:03:00.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:00 smithi083 ceph-mon[24738]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-04T16:03:00.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:00 smithi083 ceph-mon[24738]: osdmap e75: 8 total, 8 up, 8 in 2024-03-04T16:03:01.213 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:00 smithi086 ceph-mon[25584]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-04T16:03:01.213 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:00 smithi086 ceph-mon[25584]: osdmap e75: 8 total, 8 up, 8 in 2024-03-04T16:03:01.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:01 smithi086 ceph-mon[25584]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 122 KiB/s rd, 8.5 KiB/s wr, 287 op/s 2024-03-04T16:03:01.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:01 smithi086 ceph-mon[25584]: osdmap e76: 8 total, 8 up, 8 in 2024-03-04T16:03:02.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:01 smithi083 ceph-mon[24738]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 122 KiB/s rd, 8.5 KiB/s wr, 287 op/s 2024-03-04T16:03:02.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:01 smithi083 ceph-mon[24738]: osdmap e76: 8 total, 8 up, 8 in 2024-03-04T16:03:02.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:01 smithi083 ceph-mon[29452]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 122 KiB/s rd, 8.5 KiB/s wr, 287 op/s 2024-03-04T16:03:02.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:01 smithi083 ceph-mon[29452]: osdmap e76: 8 total, 8 up, 8 in 2024-03-04T16:03:02.969 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph orch apply iscsi foo u p' 2024-03-04T16:03:02.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: osdmap e77: 8 total, 8 up, 8 in 2024-03-04T16:03:02.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:02.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:02.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:02.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:02.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:02.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:02.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi083.hqmkpf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:03:02.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi083.hqmkpf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:03:02.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi083.hqmkpf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T16:03:02.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:02.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:02.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-04T16:03:02.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-04T16:03:02.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-04T16:03:02.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-04T16:03:02.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-04T16:03:02.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-04T16:03:02.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-04T16:03:02.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-04T16:03:02.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:03:02.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:03:02.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:03:02.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:03:02.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:02.992 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: osdmap e77: 8 total, 8 up, 8 in 2024-03-04T16:03:02.992 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:02.992 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:02.992 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:02.992 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:02.992 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:02.992 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:02.992 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi083.hqmkpf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:03:02.992 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi083.hqmkpf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:03:02.992 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi083.hqmkpf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T16:03:02.992 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:02.992 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:02.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-04T16:03:02.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-04T16:03:02.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-04T16:03:02.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-04T16:03:02.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-04T16:03:02.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-04T16:03:02.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-04T16:03:02.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-04T16:03:02.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:03:02.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:03:02.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:03:02.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:03:02.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: osdmap e77: 8 total, 8 up, 8 in 2024-03-04T16:03:03.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:03.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:03.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:03.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:03.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:03.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:03.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi083.hqmkpf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:03:03.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi083.hqmkpf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:03:03.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi083.hqmkpf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T16:03:03.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:03.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:03.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-04T16:03:03.237 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-04T16:03:03.237 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-04T16:03:03.237 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-04T16:03:03.237 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-04T16:03:03.237 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-04T16:03:03.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-04T16:03:03.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-04T16:03:03.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:03:03.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:03:03.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:03:03.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:03:03.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:03.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:03 smithi083 ceph-mon[24738]: Checking dashboard <-> RGW credentials 2024-03-04T16:03:03.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:03 smithi083 ceph-mon[24738]: Saving service rgw.smpl spec with placement count:2 2024-03-04T16:03:03.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:03 smithi083 ceph-mon[24738]: Deploying daemon rgw.smpl.smithi083.hqmkpf on smithi083 2024-03-04T16:03:03.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:03 smithi083 ceph-mon[24738]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 4.2 KiB/s wr, 151 op/s 2024-03-04T16:03:03.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:03 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-03-04T16:03:03.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:03 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-04T16:03:03.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:03 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-04T16:03:03.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:03 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-04T16:03:03.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:03 smithi083 ceph-mon[24738]: osdmap e78: 8 total, 8 up, 8 in 2024-03-04T16:03:03.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:03 smithi083 ceph-mon[29452]: Checking dashboard <-> RGW credentials 2024-03-04T16:03:03.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:03 smithi083 ceph-mon[29452]: Saving service rgw.smpl spec with placement count:2 2024-03-04T16:03:03.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:03 smithi083 ceph-mon[29452]: Deploying daemon rgw.smpl.smithi083.hqmkpf on smithi083 2024-03-04T16:03:03.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:03 smithi083 ceph-mon[29452]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 4.2 KiB/s wr, 151 op/s 2024-03-04T16:03:03.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:03 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-03-04T16:03:03.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:03 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-04T16:03:03.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:03 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-04T16:03:03.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:03 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-04T16:03:03.991 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:03 smithi083 ceph-mon[29452]: osdmap e78: 8 total, 8 up, 8 in 2024-03-04T16:03:04.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:03 smithi086 ceph-mon[25584]: Checking dashboard <-> RGW credentials 2024-03-04T16:03:04.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:03 smithi086 ceph-mon[25584]: Saving service rgw.smpl spec with placement count:2 2024-03-04T16:03:04.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:03 smithi086 ceph-mon[25584]: Deploying daemon rgw.smpl.smithi083.hqmkpf on smithi083 2024-03-04T16:03:04.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:03 smithi086 ceph-mon[25584]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 4.2 KiB/s wr, 151 op/s 2024-03-04T16:03:04.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:03 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-03-04T16:03:04.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:03 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-04T16:03:04.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:03 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-04T16:03:04.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:03 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-04T16:03:04.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:03 smithi086 ceph-mon[25584]: osdmap e78: 8 total, 8 up, 8 in 2024-03-04T16:03:05.827 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:05 smithi083 ceph-mon[29452]: osdmap e79: 8 total, 8 up, 8 in 2024-03-04T16:03:05.828 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:05 smithi083 ceph-mon[29452]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 166 KiB/s rd, 1023 B/s wr, 255 op/s 2024-03-04T16:03:05.828 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:05 smithi083 ceph-mon[24738]: osdmap e79: 8 total, 8 up, 8 in 2024-03-04T16:03:05.828 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:05 smithi083 ceph-mon[24738]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 166 KiB/s rd, 1023 B/s wr, 255 op/s 2024-03-04T16:03:05.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:05 smithi086 ceph-mon[25584]: osdmap e79: 8 total, 8 up, 8 in 2024-03-04T16:03:05.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:05 smithi086 ceph-mon[25584]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 166 KiB/s rd, 1023 B/s wr, 255 op/s 2024-03-04T16:03:06.067 INFO:teuthology.orchestra.run.smithi083.stdout:Scheduled iscsi.foo update... 2024-03-04T16:03:06.079 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:03:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:03:05] "GET /metrics HTTP/1.1" 200 207120 "" "Prometheus/2.33.4" 2024-03-04T16:03:07.139 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:07 smithi086 ceph-mon[25584]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:07.139 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:07 smithi086 ceph-mon[25584]: Saving service iscsi.foo spec with placement count:1 2024-03-04T16:03:07.139 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:07 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:07.139 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:07 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:07.139 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:07 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.yqigbl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:03:07.139 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:07 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.yqigbl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:03:07.139 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:07 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.yqigbl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T16:03:07.139 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:07 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:07.139 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:07 smithi086 ceph-mon[25584]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 692 B/s wr, 172 op/s 2024-03-04T16:03:07.140 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:07 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:07.140 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:07 smithi086 ceph-mon[25584]: Deploying daemon rgw.smpl.smithi086.yqigbl on smithi086 2024-03-04T16:03:07.229 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'sleep 120' 2024-03-04T16:03:07.390 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:03:07 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:03:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:03:07.455 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[24738]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:07.455 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[24738]: Saving service iscsi.foo spec with placement count:1 2024-03-04T16:03:07.455 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:07.455 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:07.455 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.yqigbl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:03:07.455 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.yqigbl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:03:07.455 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.yqigbl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T16:03:07.456 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:07.456 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[24738]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 692 B/s wr, 172 op/s 2024-03-04T16:03:07.456 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:07.456 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[24738]: Deploying daemon rgw.smpl.smithi086.yqigbl on smithi086 2024-03-04T16:03:07.456 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[29452]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:07.456 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[29452]: Saving service iscsi.foo spec with placement count:1 2024-03-04T16:03:07.456 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:07.456 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:07.456 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.yqigbl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:03:07.456 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.yqigbl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:03:07.457 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.yqigbl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T16:03:07.457 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:07.457 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[29452]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 692 B/s wr, 172 op/s 2024-03-04T16:03:07.457 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:07.457 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:07 smithi083 ceph-mon[29452]: Deploying daemon rgw.smpl.smithi086.yqigbl on smithi086 2024-03-04T16:03:09.091 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:08 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:09.192 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:08 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:09.193 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:08 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:09.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:09 smithi086 ceph-mon[25584]: pgmap v85: 161 pgs: 2 peering, 159 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 140 KiB/s rd, 741 B/s wr, 215 op/s 2024-03-04T16:03:09.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:09 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:09.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:09 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:09.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:09 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:09.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:09 smithi083 ceph-mon[29452]: pgmap v85: 161 pgs: 2 peering, 159 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 140 KiB/s rd, 741 B/s wr, 215 op/s 2024-03-04T16:03:09.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:09 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:09.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:09 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:09.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:09 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:09.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:09 smithi083 ceph-mon[24738]: pgmap v85: 161 pgs: 2 peering, 159 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 140 KiB/s rd, 741 B/s wr, 215 op/s 2024-03-04T16:03:09.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:09 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:09.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:09 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:09.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:09 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:11.373 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:11 smithi086 ceph-mon[25584]: osdmap e80: 8 total, 8 up, 8 in 2024-03-04T16:03:11.373 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:11 smithi086 ceph-mon[25584]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s rd, 718 B/s wr, 284 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:03:11.423 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:11 smithi083 ceph-mon[24738]: osdmap e80: 8 total, 8 up, 8 in 2024-03-04T16:03:11.424 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:11 smithi083 ceph-mon[24738]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s rd, 718 B/s wr, 284 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:03:11.424 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:11 smithi083 ceph-mon[29452]: osdmap e80: 8 total, 8 up, 8 in 2024-03-04T16:03:11.424 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:11 smithi083 ceph-mon[29452]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s rd, 718 B/s wr, 284 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:03:13.125 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:12 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:13.125 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:12 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:13.125 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:12 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:13.125 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:12 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:13.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:12 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:13.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:12 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:14.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:13 smithi086 ceph-mon[25584]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 126 B/s wr, 123 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:03:14.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:13 smithi086 ceph-mon[25584]: Checking dashboard <-> RGW credentials 2024-03-04T16:03:14.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:13 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:14.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:13 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:14.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:13 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:03:14.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:13 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:03:14.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:13 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-04T16:03:14.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:13 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:14.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:13 smithi083 ceph-mon[29452]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 126 B/s wr, 123 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:03:14.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:13 smithi083 ceph-mon[29452]: Checking dashboard <-> RGW credentials 2024-03-04T16:03:14.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:13 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:14.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:13 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:14.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:13 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:03:14.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:13 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:03:14.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:13 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-04T16:03:14.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:13 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:14.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:13 smithi083 ceph-mon[24738]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 126 B/s wr, 123 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:03:14.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:13 smithi083 ceph-mon[24738]: Checking dashboard <-> RGW credentials 2024-03-04T16:03:14.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:13 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:14.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:13 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:14.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:13 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:03:14.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:13 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:03:14.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:13 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-04T16:03:14.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:13 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:15.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:14 smithi086 ceph-mon[25584]: Checking pool "foo" exists for service iscsi.foo 2024-03-04T16:03:15.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:14 smithi086 ceph-mon[25584]: Deploying daemon iscsi.foo.smithi083.fnypow on smithi083 2024-03-04T16:03:15.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:14 smithi086 ceph-mon[25584]: mgrmap e20: y(active, since 2m), standbys: x 2024-03-04T16:03:15.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:14 smithi083 ceph-mon[29452]: Checking pool "foo" exists for service iscsi.foo 2024-03-04T16:03:15.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:14 smithi083 ceph-mon[29452]: Deploying daemon iscsi.foo.smithi083.fnypow on smithi083 2024-03-04T16:03:15.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:14 smithi083 ceph-mon[29452]: mgrmap e20: y(active, since 2m), standbys: x 2024-03-04T16:03:15.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:14 smithi083 ceph-mon[24738]: Checking pool "foo" exists for service iscsi.foo 2024-03-04T16:03:15.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:14 smithi083 ceph-mon[24738]: Deploying daemon iscsi.foo.smithi083.fnypow on smithi083 2024-03-04T16:03:15.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:14 smithi083 ceph-mon[24738]: mgrmap e20: y(active, since 2m), standbys: x 2024-03-04T16:03:16.019 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:03:15 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:03:15] "GET /metrics HTTP/1.1" 200 207120 "" "Prometheus/2.33.4" 2024-03-04T16:03:16.020 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:15 smithi083 ceph-mon[29452]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 204 B/s wr, 204 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:03:16.020 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:15 smithi083 ceph-mon[24738]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 204 B/s wr, 204 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:03:16.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:15 smithi086 ceph-mon[25584]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 204 B/s wr, 204 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:03:17.332 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:17 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:17.333 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:17 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:17.333 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:17 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:17.333 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:17 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/327003025' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T16:03:17.333 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:17 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3773533210' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1947529438"}]: dispatch 2024-03-04T16:03:17.333 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:17 smithi086 ceph-mon[25584]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 204 B/s wr, 204 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:03:17.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:17 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:17.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:17 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:17.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:17 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:17.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:17 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/327003025' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T16:03:17.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:17 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3773533210' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1947529438"}]: dispatch 2024-03-04T16:03:17.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:17 smithi083 ceph-mon[29452]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 204 B/s wr, 204 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:03:17.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:17 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:17.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:17 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:17.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:17 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:17.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:17 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/327003025' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T16:03:17.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:17 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3773533210' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1947529438"}]: dispatch 2024-03-04T16:03:17.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:17 smithi083 ceph-mon[24738]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 204 B/s wr, 204 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:03:17.666 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:03:17 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:03:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:03:18.372 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:18 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3773533210' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1947529438"}]': finished 2024-03-04T16:03:18.372 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:18 smithi086 ceph-mon[25584]: osdmap e81: 8 total, 8 up, 8 in 2024-03-04T16:03:18.372 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:18 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1894558763' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/3984588694"}]: dispatch 2024-03-04T16:03:18.372 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:18 smithi086 ceph-mon[25584]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/3984588694"}]: dispatch 2024-03-04T16:03:18.372 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:18 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:18.437 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:18 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3773533210' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1947529438"}]': finished 2024-03-04T16:03:18.437 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:18 smithi083 ceph-mon[24738]: osdmap e81: 8 total, 8 up, 8 in 2024-03-04T16:03:18.438 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:18 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1894558763' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/3984588694"}]: dispatch 2024-03-04T16:03:18.438 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:18 smithi083 ceph-mon[24738]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/3984588694"}]: dispatch 2024-03-04T16:03:18.438 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:18 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:18.438 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:18 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3773533210' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1947529438"}]': finished 2024-03-04T16:03:18.438 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:18 smithi083 ceph-mon[29452]: osdmap e81: 8 total, 8 up, 8 in 2024-03-04T16:03:18.438 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:18 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1894558763' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/3984588694"}]: dispatch 2024-03-04T16:03:18.438 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:18 smithi083 ceph-mon[29452]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/3984588694"}]: dispatch 2024-03-04T16:03:18.438 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:18 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:19.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:19 smithi086 ceph-mon[25584]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/3984588694"}]': finished 2024-03-04T16:03:19.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:19 smithi086 ceph-mon[25584]: osdmap e82: 8 total, 8 up, 8 in 2024-03-04T16:03:19.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:19 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3246531469' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1025905465"}]: dispatch 2024-03-04T16:03:19.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:19 smithi086 ceph-mon[25584]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1025905465"}]: dispatch 2024-03-04T16:03:19.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:19 smithi086 ceph-mon[25584]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 255 B/s wr, 131 op/s 2024-03-04T16:03:19.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:19 smithi083 ceph-mon[29452]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/3984588694"}]': finished 2024-03-04T16:03:19.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:19 smithi083 ceph-mon[29452]: osdmap e82: 8 total, 8 up, 8 in 2024-03-04T16:03:19.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:19 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3246531469' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1025905465"}]: dispatch 2024-03-04T16:03:19.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:19 smithi083 ceph-mon[29452]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1025905465"}]: dispatch 2024-03-04T16:03:19.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:19 smithi083 ceph-mon[29452]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 255 B/s wr, 131 op/s 2024-03-04T16:03:19.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:19 smithi083 ceph-mon[24738]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/3984588694"}]': finished 2024-03-04T16:03:19.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:19 smithi083 ceph-mon[24738]: osdmap e82: 8 total, 8 up, 8 in 2024-03-04T16:03:19.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:19 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3246531469' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1025905465"}]: dispatch 2024-03-04T16:03:19.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:19 smithi083 ceph-mon[24738]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1025905465"}]: dispatch 2024-03-04T16:03:19.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:19 smithi083 ceph-mon[24738]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 255 B/s wr, 131 op/s 2024-03-04T16:03:20.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1025905465"}]': finished 2024-03-04T16:03:20.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: osdmap e83: 8 total, 8 up, 8 in 2024-03-04T16:03:20.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:20.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1312507343' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/645025785"}]: dispatch 2024-03-04T16:03:20.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/645025785"}]: dispatch 2024-03-04T16:03:20.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:20.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:20.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-04T16:03:20.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-04T16:03:20.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: Adding iSCSI gateway http://:@172.21.15.83:5000 to Dashboard 2024-03-04T16:03:20.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-04T16:03:20.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-04T16:03:20.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:20.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi083"}]: dispatch 2024-03-04T16:03:20.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi083"}]: dispatch 2024-03-04T16:03:20.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:20.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:20.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:20.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: Checking dashboard <-> RGW credentials 2024-03-04T16:03:20.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:20.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1025905465"}]': finished 2024-03-04T16:03:20.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: osdmap e83: 8 total, 8 up, 8 in 2024-03-04T16:03:20.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:20.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1312507343' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/645025785"}]: dispatch 2024-03-04T16:03:20.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/645025785"}]: dispatch 2024-03-04T16:03:20.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:20.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:20.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-04T16:03:20.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-04T16:03:20.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: Adding iSCSI gateway http://:@172.21.15.83:5000 to Dashboard 2024-03-04T16:03:20.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-04T16:03:20.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-04T16:03:20.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:20.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi083"}]: dispatch 2024-03-04T16:03:20.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi083"}]: dispatch 2024-03-04T16:03:20.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:20.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:20.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:20.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: Checking dashboard <-> RGW credentials 2024-03-04T16:03:20.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:20 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:20.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1025905465"}]': finished 2024-03-04T16:03:20.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: osdmap e83: 8 total, 8 up, 8 in 2024-03-04T16:03:20.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:20.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1312507343' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/645025785"}]: dispatch 2024-03-04T16:03:20.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/645025785"}]: dispatch 2024-03-04T16:03:20.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:20.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:20.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-04T16:03:20.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-04T16:03:20.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: Adding iSCSI gateway http://:@172.21.15.83:5000 to Dashboard 2024-03-04T16:03:20.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-04T16:03:20.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-04T16:03:20.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:20.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi083"}]: dispatch 2024-03-04T16:03:20.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi083"}]: dispatch 2024-03-04T16:03:20.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:20.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:20.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:20.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: Checking dashboard <-> RGW credentials 2024-03-04T16:03:20.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:20 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:03:21.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:21 smithi086 ceph-mon[25584]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/645025785"}]': finished 2024-03-04T16:03:21.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:21 smithi086 ceph-mon[25584]: osdmap e84: 8 total, 8 up, 8 in 2024-03-04T16:03:21.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:21 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1375013195' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/1610714353"}]: dispatch 2024-03-04T16:03:21.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:21 smithi086 ceph-mon[25584]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T16:03:21.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:21 smithi083 ceph-mon[24738]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/645025785"}]': finished 2024-03-04T16:03:21.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:21 smithi083 ceph-mon[24738]: osdmap e84: 8 total, 8 up, 8 in 2024-03-04T16:03:21.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:21 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1375013195' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/1610714353"}]: dispatch 2024-03-04T16:03:21.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:21 smithi083 ceph-mon[24738]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T16:03:21.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:21 smithi083 ceph-mon[29452]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/645025785"}]': finished 2024-03-04T16:03:21.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:21 smithi083 ceph-mon[29452]: osdmap e84: 8 total, 8 up, 8 in 2024-03-04T16:03:21.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:21 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1375013195' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/1610714353"}]: dispatch 2024-03-04T16:03:21.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:21 smithi083 ceph-mon[29452]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T16:03:22.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:22 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1375013195' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/1610714353"}]': finished 2024-03-04T16:03:22.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:22 smithi086 ceph-mon[25584]: osdmap e85: 8 total, 8 up, 8 in 2024-03-04T16:03:22.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:22 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3961333088' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/1610714353"}]: dispatch 2024-03-04T16:03:22.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:22 smithi086 ceph-mon[25584]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/1610714353"}]: dispatch 2024-03-04T16:03:22.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:22 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1375013195' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/1610714353"}]': finished 2024-03-04T16:03:22.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:22 smithi083 ceph-mon[24738]: osdmap e85: 8 total, 8 up, 8 in 2024-03-04T16:03:22.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:22 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3961333088' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/1610714353"}]: dispatch 2024-03-04T16:03:22.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:22 smithi083 ceph-mon[24738]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/1610714353"}]: dispatch 2024-03-04T16:03:22.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:22 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1375013195' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/1610714353"}]': finished 2024-03-04T16:03:22.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:22 smithi083 ceph-mon[29452]: osdmap e85: 8 total, 8 up, 8 in 2024-03-04T16:03:22.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:22 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3961333088' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/1610714353"}]: dispatch 2024-03-04T16:03:22.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:22 smithi083 ceph-mon[29452]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/1610714353"}]: dispatch 2024-03-04T16:03:23.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:23 smithi086 ceph-mon[25584]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/1610714353"}]': finished 2024-03-04T16:03:23.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:23 smithi086 ceph-mon[25584]: osdmap e86: 8 total, 8 up, 8 in 2024-03-04T16:03:23.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:23 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/4174964992' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1017961836"}]: dispatch 2024-03-04T16:03:23.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:23 smithi086 ceph-mon[25584]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:03:23.738 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:03:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:03:23.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-03-04T16:03:23.738 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:03:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:03:23.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.83:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.83 because it doesn't contain any IP SANs" 2024-03-04T16:03:23.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:23 smithi083 ceph-mon[24738]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/1610714353"}]': finished 2024-03-04T16:03:23.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:23 smithi083 ceph-mon[24738]: osdmap e86: 8 total, 8 up, 8 in 2024-03-04T16:03:23.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:23 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/4174964992' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1017961836"}]: dispatch 2024-03-04T16:03:23.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:23 smithi083 ceph-mon[24738]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:03:23.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:23 smithi083 ceph-mon[29452]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/1610714353"}]': finished 2024-03-04T16:03:23.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:23 smithi083 ceph-mon[29452]: osdmap e86: 8 total, 8 up, 8 in 2024-03-04T16:03:23.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:23 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/4174964992' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1017961836"}]: dispatch 2024-03-04T16:03:23.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:23 smithi083 ceph-mon[29452]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:03:24.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:24 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/4174964992' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1017961836"}]': finished 2024-03-04T16:03:24.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:24 smithi086 ceph-mon[25584]: osdmap e87: 8 total, 8 up, 8 in 2024-03-04T16:03:24.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:24 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/420455928' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1054338912"}]: dispatch 2024-03-04T16:03:24.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:24 smithi086 ceph-mon[25584]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1054338912"}]: dispatch 2024-03-04T16:03:24.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:24 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/4174964992' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1017961836"}]': finished 2024-03-04T16:03:24.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:24 smithi083 ceph-mon[29452]: osdmap e87: 8 total, 8 up, 8 in 2024-03-04T16:03:24.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:24 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/420455928' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1054338912"}]: dispatch 2024-03-04T16:03:24.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:24 smithi083 ceph-mon[29452]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1054338912"}]: dispatch 2024-03-04T16:03:24.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:24 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/4174964992' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1017961836"}]': finished 2024-03-04T16:03:24.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:24 smithi083 ceph-mon[24738]: osdmap e87: 8 total, 8 up, 8 in 2024-03-04T16:03:24.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:24 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/420455928' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1054338912"}]: dispatch 2024-03-04T16:03:24.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:24 smithi083 ceph-mon[24738]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1054338912"}]: dispatch 2024-03-04T16:03:25.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:25 smithi086 ceph-mon[25584]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1054338912"}]': finished 2024-03-04T16:03:25.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:25 smithi086 ceph-mon[25584]: osdmap e88: 8 total, 8 up, 8 in 2024-03-04T16:03:25.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:25 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/2717593840' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1893644918"}]: dispatch 2024-03-04T16:03:25.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:25 smithi086 ceph-mon[25584]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-04T16:03:25.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:25 smithi083 ceph-mon[29452]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1054338912"}]': finished 2024-03-04T16:03:25.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:25 smithi083 ceph-mon[29452]: osdmap e88: 8 total, 8 up, 8 in 2024-03-04T16:03:25.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:25 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/2717593840' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1893644918"}]: dispatch 2024-03-04T16:03:25.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:25 smithi083 ceph-mon[29452]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-04T16:03:25.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:25 smithi083 ceph-mon[24738]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1054338912"}]': finished 2024-03-04T16:03:25.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:25 smithi083 ceph-mon[24738]: osdmap e88: 8 total, 8 up, 8 in 2024-03-04T16:03:25.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:25 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2717593840' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1893644918"}]: dispatch 2024-03-04T16:03:25.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:25 smithi083 ceph-mon[24738]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-04T16:03:26.214 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:03:25 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:03:25] "GET /metrics HTTP/1.1" 200 214935 "" "Prometheus/2.33.4" 2024-03-04T16:03:26.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:26 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/2717593840' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1893644918"}]': finished 2024-03-04T16:03:26.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:26 smithi083 ceph-mon[29452]: osdmap e89: 8 total, 8 up, 8 in 2024-03-04T16:03:26.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:26 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/808979027' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/2606489355"}]: dispatch 2024-03-04T16:03:26.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:26 smithi083 ceph-mon[29452]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/2606489355"}]: dispatch 2024-03-04T16:03:26.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:26 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2717593840' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1893644918"}]': finished 2024-03-04T16:03:26.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:26 smithi083 ceph-mon[24738]: osdmap e89: 8 total, 8 up, 8 in 2024-03-04T16:03:26.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:26 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/808979027' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/2606489355"}]: dispatch 2024-03-04T16:03:26.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:26 smithi083 ceph-mon[24738]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/2606489355"}]: dispatch 2024-03-04T16:03:26.635 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:26 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/2717593840' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1893644918"}]': finished 2024-03-04T16:03:26.636 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:26 smithi086 ceph-mon[25584]: osdmap e89: 8 total, 8 up, 8 in 2024-03-04T16:03:26.636 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:26 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/808979027' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/2606489355"}]: dispatch 2024-03-04T16:03:26.636 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:26 smithi086 ceph-mon[25584]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/2606489355"}]: dispatch 2024-03-04T16:03:27.647 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:27 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:27.648 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:27 smithi086 ceph-mon[25584]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/2606489355"}]': finished 2024-03-04T16:03:27.648 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:27 smithi086 ceph-mon[25584]: osdmap e90: 8 total, 8 up, 8 in 2024-03-04T16:03:27.648 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:27 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/2164588498' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/2606489355"}]: dispatch 2024-03-04T16:03:27.648 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:27 smithi086 ceph-mon[25584]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/2606489355"}]: dispatch 2024-03-04T16:03:27.648 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:27 smithi086 ceph-mon[25584]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-04T16:03:27.648 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:03:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:03:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:03:27.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:27 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:27.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:27 smithi083 ceph-mon[24738]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/2606489355"}]': finished 2024-03-04T16:03:27.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:27 smithi083 ceph-mon[24738]: osdmap e90: 8 total, 8 up, 8 in 2024-03-04T16:03:27.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:27 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2164588498' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/2606489355"}]: dispatch 2024-03-04T16:03:27.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:27 smithi083 ceph-mon[24738]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/2606489355"}]: dispatch 2024-03-04T16:03:27.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:27 smithi083 ceph-mon[24738]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-04T16:03:27.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:27 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:27.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:27 smithi083 ceph-mon[29452]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/2606489355"}]': finished 2024-03-04T16:03:27.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:27 smithi083 ceph-mon[29452]: osdmap e90: 8 total, 8 up, 8 in 2024-03-04T16:03:27.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:27 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/2164588498' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/2606489355"}]: dispatch 2024-03-04T16:03:27.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:27 smithi083 ceph-mon[29452]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/2606489355"}]: dispatch 2024-03-04T16:03:27.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:27 smithi083 ceph-mon[29452]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-04T16:03:28.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:28 smithi086 ceph-mon[25584]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/2606489355"}]': finished 2024-03-04T16:03:28.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:28 smithi086 ceph-mon[25584]: osdmap e91: 8 total, 8 up, 8 in 2024-03-04T16:03:28.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:28 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3491959005' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/645025785"}]: dispatch 2024-03-04T16:03:28.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:28 smithi083 ceph-mon[24738]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/2606489355"}]': finished 2024-03-04T16:03:28.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:28 smithi083 ceph-mon[24738]: osdmap e91: 8 total, 8 up, 8 in 2024-03-04T16:03:28.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:28 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3491959005' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/645025785"}]: dispatch 2024-03-04T16:03:28.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:28 smithi083 ceph-mon[29452]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/2606489355"}]': finished 2024-03-04T16:03:28.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:28 smithi083 ceph-mon[29452]: osdmap e91: 8 total, 8 up, 8 in 2024-03-04T16:03:28.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:28 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3491959005' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/645025785"}]: dispatch 2024-03-04T16:03:29.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:29 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3491959005' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/645025785"}]': finished 2024-03-04T16:03:29.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:29 smithi086 ceph-mon[25584]: osdmap e92: 8 total, 8 up, 8 in 2024-03-04T16:03:29.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:29 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3771198625' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2983002978"}]: dispatch 2024-03-04T16:03:29.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:29 smithi086 ceph-mon[25584]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:29.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:29 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3491959005' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/645025785"}]': finished 2024-03-04T16:03:29.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:29 smithi083 ceph-mon[29452]: osdmap e92: 8 total, 8 up, 8 in 2024-03-04T16:03:29.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:29 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3771198625' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2983002978"}]: dispatch 2024-03-04T16:03:29.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:29 smithi083 ceph-mon[29452]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:29.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:29 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3491959005' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/645025785"}]': finished 2024-03-04T16:03:29.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:29 smithi083 ceph-mon[24738]: osdmap e92: 8 total, 8 up, 8 in 2024-03-04T16:03:29.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:29 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3771198625' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2983002978"}]: dispatch 2024-03-04T16:03:29.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:29 smithi083 ceph-mon[24738]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:30.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:30 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3771198625' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2983002978"}]': finished 2024-03-04T16:03:30.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:30 smithi086 ceph-mon[25584]: osdmap e93: 8 total, 8 up, 8 in 2024-03-04T16:03:30.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:30 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1383285540' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/489466195"}]: dispatch 2024-03-04T16:03:30.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:30 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3771198625' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2983002978"}]': finished 2024-03-04T16:03:30.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:30 smithi083 ceph-mon[29452]: osdmap e93: 8 total, 8 up, 8 in 2024-03-04T16:03:30.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:30 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1383285540' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/489466195"}]: dispatch 2024-03-04T16:03:30.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:30 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3771198625' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2983002978"}]': finished 2024-03-04T16:03:30.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:30 smithi083 ceph-mon[24738]: osdmap e93: 8 total, 8 up, 8 in 2024-03-04T16:03:30.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:30 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1383285540' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/489466195"}]: dispatch 2024-03-04T16:03:31.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:31 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1383285540' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/489466195"}]': finished 2024-03-04T16:03:31.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:31 smithi086 ceph-mon[25584]: osdmap e94: 8 total, 8 up, 8 in 2024-03-04T16:03:31.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:31 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/41340246' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1474787565"}]: dispatch 2024-03-04T16:03:31.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:31 smithi086 ceph-mon[25584]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:31.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:31 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1383285540' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/489466195"}]': finished 2024-03-04T16:03:31.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:31 smithi083 ceph-mon[24738]: osdmap e94: 8 total, 8 up, 8 in 2024-03-04T16:03:31.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:31 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/41340246' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1474787565"}]: dispatch 2024-03-04T16:03:31.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:31 smithi083 ceph-mon[24738]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:31.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:31 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1383285540' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/489466195"}]': finished 2024-03-04T16:03:31.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:31 smithi083 ceph-mon[29452]: osdmap e94: 8 total, 8 up, 8 in 2024-03-04T16:03:31.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:31 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/41340246' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1474787565"}]: dispatch 2024-03-04T16:03:31.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:31 smithi083 ceph-mon[29452]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:32.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:32 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/41340246' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1474787565"}]': finished 2024-03-04T16:03:32.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:32 smithi086 ceph-mon[25584]: osdmap e95: 8 total, 8 up, 8 in 2024-03-04T16:03:32.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:32 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/2289763122' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/106501873"}]: dispatch 2024-03-04T16:03:32.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:32 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/41340246' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1474787565"}]': finished 2024-03-04T16:03:32.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:32 smithi083 ceph-mon[24738]: osdmap e95: 8 total, 8 up, 8 in 2024-03-04T16:03:32.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:32 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2289763122' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/106501873"}]: dispatch 2024-03-04T16:03:32.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:32 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/41340246' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1474787565"}]': finished 2024-03-04T16:03:32.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:32 smithi083 ceph-mon[29452]: osdmap e95: 8 total, 8 up, 8 in 2024-03-04T16:03:32.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:32 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/2289763122' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/106501873"}]: dispatch 2024-03-04T16:03:33.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:33 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/2289763122' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/106501873"}]': finished 2024-03-04T16:03:33.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:33 smithi086 ceph-mon[25584]: osdmap e96: 8 total, 8 up, 8 in 2024-03-04T16:03:33.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:33 smithi086 ceph-mon[25584]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:03:33.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:33 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2289763122' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/106501873"}]': finished 2024-03-04T16:03:33.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:33 smithi083 ceph-mon[24738]: osdmap e96: 8 total, 8 up, 8 in 2024-03-04T16:03:33.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:33 smithi083 ceph-mon[24738]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:03:33.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:33 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/2289763122' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/106501873"}]': finished 2024-03-04T16:03:33.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:33 smithi083 ceph-mon[29452]: osdmap e96: 8 total, 8 up, 8 in 2024-03-04T16:03:33.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:33 smithi083 ceph-mon[29452]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:03:33.739 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:03:33 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=error ts=2024-03-04T16:03:33.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.83:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.83 because it doesn't contain any IP SANs" 2024-03-04T16:03:33.739 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:03:33 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:03:33.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-03-04T16:03:33.739 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:03:33 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:03:33.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.83:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.83 because it doesn't contain any IP SANs" 2024-03-04T16:03:36.224 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:35 smithi083 ceph-mon[29452]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-04T16:03:36.225 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:35 smithi083 ceph-mon[24738]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-04T16:03:36.225 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:03:35 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:03:35] "GET /metrics HTTP/1.1" 200 214960 "" "Prometheus/2.33.4" 2024-03-04T16:03:36.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:35 smithi086 ceph-mon[25584]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-04T16:03:37.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:03:37 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:03:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:03:38.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:37 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:38.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:37 smithi086 ceph-mon[25584]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 793 B/s rd, 0 op/s 2024-03-04T16:03:38.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:37 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:38.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:37 smithi083 ceph-mon[24738]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 793 B/s rd, 0 op/s 2024-03-04T16:03:38.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:37 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:38.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:37 smithi083 ceph-mon[29452]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 793 B/s rd, 0 op/s 2024-03-04T16:03:40.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:39 smithi086 ceph-mon[25584]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:40.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:39 smithi083 ceph-mon[24738]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:40.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:39 smithi083 ceph-mon[29452]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:41.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:41 smithi086 ceph-mon[25584]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:03:41.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:41 smithi083 ceph-mon[24738]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:03:41.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:41 smithi083 ceph-mon[29452]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:03:43.884 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:03:43 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=error ts=2024-03-04T16:03:43.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.83:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.83 because it doesn't contain any IP SANs" 2024-03-04T16:03:44.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:43 smithi086 ceph-mon[25584]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 981 B/s rd, 0 op/s 2024-03-04T16:03:44.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:43 smithi083 ceph-mon[29452]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 981 B/s rd, 0 op/s 2024-03-04T16:03:44.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:43 smithi083 ceph-mon[24738]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 981 B/s rd, 0 op/s 2024-03-04T16:03:46.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:45 smithi086 ceph-mon[25584]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:46.235 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:45 smithi083 ceph-mon[29452]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:46.235 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:45 smithi083 ceph-mon[24738]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:46.236 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:03:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:03:45] "GET /metrics HTTP/1.1" 200 214960 "" "Prometheus/2.33.4" 2024-03-04T16:03:47.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:03:47 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:03:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:03:48.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:47 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:48.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:47 smithi086 ceph-mon[25584]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:03:48.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:47 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:48.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:47 smithi083 ceph-mon[24738]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:03:48.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:47 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:48.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:47 smithi083 ceph-mon[29452]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:03:50.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:49 smithi086 ceph-mon[25584]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:50.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:49 smithi083 ceph-mon[29452]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:50.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:49 smithi083 ceph-mon[24738]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:51.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:51 smithi086 ceph-mon[25584]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:03:51.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:51 smithi083 ceph-mon[24738]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:03:51.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:51 smithi083 ceph-mon[29452]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:03:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:53 smithi086 ceph-mon[25584]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:03:54.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:53 smithi083 ceph-mon[24738]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:03:54.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:53 smithi083 ceph-mon[29452]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:03:56.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:55 smithi086 ceph-mon[25584]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:56.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:55 smithi083 ceph-mon[24738]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:56.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:03:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:03:55] "GET /metrics HTTP/1.1" 200 214908 "" "Prometheus/2.33.4" 2024-03-04T16:03:56.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:55 smithi083 ceph-mon[29452]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:57.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:03:57 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:03:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:03:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:57 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:58.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:57 smithi086 ceph-mon[25584]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:03:58.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:57 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:58.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:57 smithi083 ceph-mon[24738]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:03:58.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:57 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:58.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:57 smithi083 ceph-mon[29452]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:00.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:03:59 smithi086 ceph-mon[25584]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:00.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:03:59 smithi083 ceph-mon[24738]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:00.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:03:59 smithi083 ceph-mon[29452]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:01.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:01 smithi086 ceph-mon[25584]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:01.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:01 smithi083 ceph-mon[24738]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:01.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:01 smithi083 ceph-mon[29452]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:02 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:04:03.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:02 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:04:03.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:04:03.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:04:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:02 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:04:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:02 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:04:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:04:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:04:03.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:02 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:04:03.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:02 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:04:03.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:04:03.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:04:04.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:03 smithi086 ceph-mon[25584]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:04.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:03 smithi083 ceph-mon[24738]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:04.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:03 smithi083 ceph-mon[29452]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:06.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:05 smithi086 ceph-mon[25584]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:06.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:05 smithi083 ceph-mon[24738]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:06.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:05 smithi083 ceph-mon[29452]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:06.239 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:04:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:04:05] "GET /metrics HTTP/1.1" 200 214857 "" "Prometheus/2.33.4" 2024-03-04T16:04:07.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:04:07 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:04:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:04:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:07 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:07 smithi086 ceph-mon[25584]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:08.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:07 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:08.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:07 smithi083 ceph-mon[24738]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:08.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:07 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:08.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:07 smithi083 ceph-mon[29452]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:09 smithi086 ceph-mon[25584]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:10.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:09 smithi083 ceph-mon[24738]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:10.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:09 smithi083 ceph-mon[29452]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:11.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:11 smithi086 ceph-mon[25584]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:11.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:11 smithi083 ceph-mon[24738]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:11.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:11 smithi083 ceph-mon[29452]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:14.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:13 smithi086 ceph-mon[25584]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:14.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:13 smithi083 ceph-mon[29452]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:14.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:13 smithi083 ceph-mon[24738]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:16.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:15 smithi086 ceph-mon[25584]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:16.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:15 smithi083 ceph-mon[24738]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:16.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:04:15 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:04:15] "GET /metrics HTTP/1.1" 200 214857 "" "Prometheus/2.33.4" 2024-03-04T16:04:16.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:15 smithi083 ceph-mon[29452]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:17.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:04:17 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:04:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:04:18.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:17 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:18.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:17 smithi086 ceph-mon[25584]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:18.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:17 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:18.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:17 smithi083 ceph-mon[29452]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:18.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:17 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:18.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:17 smithi083 ceph-mon[24738]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:20.154 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:19 smithi083 ceph-mon[24738]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:20.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:19 smithi083 ceph-mon[29452]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:20.158 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:19 smithi086 ceph-mon[25584]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:21.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:20 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:04:21.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:20 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:04:21.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:20 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:04:21.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:20 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:04:21.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:20 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:04:21.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:20 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:04:21.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:20 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:04:21.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:20 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:04:21.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:20 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:04:22.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:21 smithi086 ceph-mon[25584]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:22.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:21 smithi083 ceph-mon[29452]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:22.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:21 smithi083 ceph-mon[24738]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:24.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:23 smithi086 ceph-mon[25584]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:24.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:23 smithi083 ceph-mon[29452]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:24.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:23 smithi083 ceph-mon[24738]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:26.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:25 smithi086 ceph-mon[25584]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:26.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:25 smithi083 ceph-mon[24738]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:26.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:25 smithi083 ceph-mon[29452]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:26.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:04:25 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:04:25] "GET /metrics HTTP/1.1" 200 214850 "" "Prometheus/2.33.4" 2024-03-04T16:04:27.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:04:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:04:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:04:28.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:27 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:28.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:27 smithi086 ceph-mon[25584]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:28.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:27 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:28.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:27 smithi083 ceph-mon[24738]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:28.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:27 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:28.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:27 smithi083 ceph-mon[29452]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:30.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:29 smithi086 ceph-mon[25584]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:30.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:29 smithi083 ceph-mon[24738]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:30.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:29 smithi083 ceph-mon[29452]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:31.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:31 smithi086 ceph-mon[25584]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:31.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:31 smithi083 ceph-mon[29452]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:31.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:31 smithi083 ceph-mon[24738]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:34.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:33 smithi086 ceph-mon[25584]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:34.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:33 smithi083 ceph-mon[29452]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:34.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:33 smithi083 ceph-mon[24738]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:36.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:35 smithi086 ceph-mon[25584]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:36.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:35 smithi083 ceph-mon[24738]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:36.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:04:35 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:04:35] "GET /metrics HTTP/1.1" 200 214849 "" "Prometheus/2.33.4" 2024-03-04T16:04:36.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:35 smithi083 ceph-mon[29452]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:37.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:04:37 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:04:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:04:38.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:37 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:38.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:37 smithi086 ceph-mon[25584]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:38.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:37 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:38.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:37 smithi083 ceph-mon[29452]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:38.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:37 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:38.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:37 smithi083 ceph-mon[24738]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:40.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:39 smithi086 ceph-mon[25584]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:40.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:39 smithi083 ceph-mon[24738]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:40.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:39 smithi083 ceph-mon[29452]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:41.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:41 smithi086 ceph-mon[25584]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:41.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:41 smithi083 ceph-mon[24738]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:41.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:41 smithi083 ceph-mon[29452]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:44.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:43 smithi086 ceph-mon[25584]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:44.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:43 smithi083 ceph-mon[24738]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:44.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:43 smithi083 ceph-mon[29452]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:46.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:45 smithi086 ceph-mon[25584]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:46.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:45 smithi083 ceph-mon[24738]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:46.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:04:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:04:45] "GET /metrics HTTP/1.1" 200 214849 "" "Prometheus/2.33.4" 2024-03-04T16:04:46.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:45 smithi083 ceph-mon[29452]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:47.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:04:47 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:04:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:04:48.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:47 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:48.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:47 smithi086 ceph-mon[25584]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:48.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:47 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:48.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:47 smithi083 ceph-mon[24738]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:48.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:47 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:48.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:47 smithi083 ceph-mon[29452]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:50.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:49 smithi086 ceph-mon[25584]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:50.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:49 smithi083 ceph-mon[24738]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:50.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:49 smithi083 ceph-mon[29452]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:51.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:51 smithi086 ceph-mon[25584]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:51.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:51 smithi083 ceph-mon[24738]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:51.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:51 smithi083 ceph-mon[29452]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:53 smithi086 ceph-mon[25584]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:54.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:53 smithi083 ceph-mon[24738]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:54.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:53 smithi083 ceph-mon[29452]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:56.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:55 smithi086 ceph-mon[25584]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:56.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:55 smithi083 ceph-mon[24738]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:56.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:04:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:04:55] "GET /metrics HTTP/1.1" 200 214847 "" "Prometheus/2.33.4" 2024-03-04T16:04:56.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:55 smithi083 ceph-mon[29452]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:57.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:04:57 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:04:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:04:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:57 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:57 smithi086 ceph-mon[25584]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:58.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:57 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:58.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:57 smithi083 ceph-mon[24738]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:58.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:57 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:58.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:57 smithi083 ceph-mon[29452]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:00.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:04:59 smithi086 ceph-mon[25584]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:00.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:04:59 smithi083 ceph-mon[24738]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:00.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:04:59 smithi083 ceph-mon[29452]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:01.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:01 smithi086 ceph-mon[25584]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:01.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:01 smithi083 ceph-mon[24738]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:01.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:01 smithi083 ceph-mon[29452]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:02 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:05:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:02 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:05:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:05:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:05:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:02 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:05:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:02 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:05:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:05:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:05:03.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:02 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:05:03.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:02 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:05:03.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:05:03.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:05:04.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:03 smithi086 ceph-mon[25584]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:04.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:03 smithi083 ceph-mon[24738]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:04.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:03 smithi083 ceph-mon[29452]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:06.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:05 smithi086 ceph-mon[25584]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:06.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:05 smithi083 ceph-mon[24738]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:06.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:05:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:05:05] "GET /metrics HTTP/1.1" 200 214872 "" "Prometheus/2.33.4" 2024-03-04T16:05:06.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:05 smithi083 ceph-mon[29452]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:07.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:05:07 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:05:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:05:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:07 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:07 smithi086 ceph-mon[25584]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:08.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:07 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:08.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:07 smithi083 ceph-mon[24738]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:08.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:07 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:08.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:07 smithi083 ceph-mon[29452]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:09.754 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-04T16:05:09.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:09 smithi083 ceph-mon[24738]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:09.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:09 smithi083 ceph-mon[29452]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:09 smithi086 ceph-mon[25584]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:11.263 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:11 smithi083 ceph-mon[29452]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:11.264 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:11 smithi083 ceph-mon[24738]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:11.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:11 smithi086 ceph-mon[25584]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:12.371 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-04T16:05:14.196 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:13 smithi083 ceph-mon[24738]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:14.196 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:13 smithi083 ceph-mon[29452]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:14.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:13 smithi086 ceph-mon[25584]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:15.040 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-04T16:05:15.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:15 smithi083 ceph-mon[29452]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:15.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:15 smithi083 ceph-mon[24738]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:15.989 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:05:15 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:05:15] "GET /metrics HTTP/1.1" 200 214872 "" "Prometheus/2.33.4" 2024-03-04T16:05:16.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:15 smithi086 ceph-mon[25584]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:17.612 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-04T16:05:17.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:05:17 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:05:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:05:18.218 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:17 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:18.218 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:17 smithi083 ceph-mon[29452]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:18.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:17 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:18.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:17 smithi083 ceph-mon[24738]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:18.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:17 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:18.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:17 smithi086 ceph-mon[25584]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:19.477 INFO:teuthology.orchestra.run.smithi083.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:05:20.143 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-04T16:05:20.154 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi083.front.sepia.ceph.com 2024-03-04T16:05:20.154 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-04T16:05:20.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:19 smithi083 ceph-mon[29452]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:20.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:19 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:05:20.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:19 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:05:20.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:19 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:05:20.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:19 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:05:20.178 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:19 smithi083 ceph-mon[24738]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:20.179 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:19 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:05:20.179 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:19 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:05:20.179 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:19 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:05:20.179 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:19 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:05:20.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:19 smithi086 ceph-mon[25584]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:20.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:19 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:05:20.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:19 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:05:20.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:19 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:05:20.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:19 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:05:21.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:20 smithi086 ceph-mon[25584]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:21.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:20 smithi086 ceph-mon[25584]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:05:21.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:20 smithi086 ceph-mon[25584]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:05:21.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:20 smithi083 ceph-mon[24738]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:21.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:20 smithi083 ceph-mon[24738]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:05:21.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:20 smithi083 ceph-mon[24738]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:05:21.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:20 smithi083 ceph-mon[29452]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:21.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:20 smithi083 ceph-mon[29452]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:05:21.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:20 smithi083 ceph-mon[29452]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:05:22.201 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:21 smithi083 ceph-mon[24738]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:22.202 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:21 smithi083 ceph-mon[29452]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:22.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:21 smithi086 ceph-mon[25584]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:22.444 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:05:23.085 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:05:23.085 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (2m) 2m ago 3m 15.5M - ba2b418f427c 8a58c6dd3c10 2024-03-04T16:05:23.085 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (2m) 2m ago 2m 46.0M - 8.3.5 dad864ee21e9 9c0f7c6afe4b 2024-03-04T16:05:23.085 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 running (2m) 2m ago 2m 68.0M - 3.5 e1d6a67b021e 553466e95b30 2024-03-04T16:05:23.085 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443 running (7m) 2m ago 7m 411M - 17.2.0 e1d6a67b021e 799a7f926005 2024-03-04T16:05:23.085 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:9283 running (9m) 2m ago 9m 456M - 17.2.0 e1d6a67b021e 5af09a7ff74e 2024-03-04T16:05:23.085 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (9m) 2m ago 9m 46.8M 2048M 17.2.0 e1d6a67b021e d2ba878ed364 2024-03-04T16:05:23.086 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (7m) 2m ago 7m 38.9M 2048M 17.2.0 e1d6a67b021e d1b972ebdd2a 2024-03-04T16:05:23.086 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (8m) 2m ago 8m 38.5M 2048M 17.2.0 e1d6a67b021e e74b0ab73a80 2024-03-04T16:05:23.086 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (3m) 2m ago 3m 14.2M - 1dbe0e931976 7eb96ca65b53 2024-03-04T16:05:23.086 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (3m) 2m ago 3m 14.4M - 1dbe0e931976 9091c36c27a4 2024-03-04T16:05:23.086 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (7m) 2m ago 7m 49.3M 4016M 17.2.0 e1d6a67b021e 56679a9f787b 2024-03-04T16:05:23.086 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (7m) 2m ago 7m 52.2M 4016M 17.2.0 e1d6a67b021e c1070c1a315a 2024-03-04T16:05:23.086 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (6m) 2m ago 6m 45.2M 4016M 17.2.0 e1d6a67b021e 2b913c19f581 2024-03-04T16:05:23.086 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (6m) 2m ago 6m 47.1M 4016M 17.2.0 e1d6a67b021e 9f9a4e2ec5da 2024-03-04T16:05:23.086 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (5m) 2m ago 5m 48.2M 4272M 17.2.0 e1d6a67b021e b3a97f83f1c7 2024-03-04T16:05:23.086 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (5m) 2m ago 5m 45.7M 4272M 17.2.0 e1d6a67b021e b82ca42b7353 2024-03-04T16:05:23.086 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (5m) 2m ago 5m 45.0M 4272M 17.2.0 e1d6a67b021e 18cedf14321d 2024-03-04T16:05:23.086 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (4m) 2m ago 4m 49.7M 4272M 17.2.0 e1d6a67b021e 390e1798c738 2024-03-04T16:05:23.086 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 running (2m) 2m ago 4m 43.2M - 514e6a882f6e 82e50b4a8b86 2024-03-04T16:05:23.086 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (2m) 2m ago 2m 70.5M - 17.2.0 e1d6a67b021e 65f14042f354 2024-03-04T16:05:23.086 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (2m) 2m ago 2m 71.3M - 17.2.0 e1d6a67b021e 78175cc6dc5d 2024-03-04T16:05:23.087 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (2m) 2m ago 2m 68.8M - 17.2.0 e1d6a67b021e 01dcf858bd62 2024-03-04T16:05:23.087 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (2m) 2m ago 2m 73.4M - 17.2.0 e1d6a67b021e 8f70a5ce9f0d 2024-03-04T16:05:23.456 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:05:23.456 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:05:23.456 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-04T16:05:23.456 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:05:23.456 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:05:23.456 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-04T16:05:23.456 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:05:23.456 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:05:23.456 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-04T16:05:23.456 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:05:23.456 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": {}, 2024-03-04T16:05:23.457 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:05:23.457 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-04T16:05:23.457 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:05:23.457 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:05:23.457 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-03-04T16:05:23.457 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:05:23.457 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:05:23.809 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:05:23.809 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:05:23.809 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:05:23.809 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [], 2024-03-04T16:05:23.809 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "", 2024-03-04T16:05:23.810 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 image" 2024-03-04T16:05:23.810 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:05:24.191 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_OK 2024-03-04T16:05:24.192 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:23 smithi083 ceph-mon[29452]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:24.192 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:23 smithi083 ceph-mon[29452]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:24.192 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:23 smithi083 ceph-mon[29452]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:24.192 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:23 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/618153786' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:05:24.195 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:23 smithi083 ceph-mon[24738]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:24.195 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:23 smithi083 ceph-mon[24738]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:24.195 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:23 smithi083 ceph-mon[24738]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:24.195 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:23 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/618153786' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:05:24.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:23 smithi086 ceph-mon[25584]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:24.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:23 smithi086 ceph-mon[25584]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:24.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:23 smithi086 ceph-mon[25584]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:24.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:23 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/618153786' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:05:25.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:24 smithi086 ceph-mon[25584]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:25.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:24 smithi086 ceph-mon[25584]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:25.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:24 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/4009640281' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:05:25.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:24 smithi083 ceph-mon[24738]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:25.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:24 smithi083 ceph-mon[24738]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:25.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:24 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/4009640281' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:05:25.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:24 smithi083 ceph-mon[29452]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:25.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:24 smithi083 ceph-mon[29452]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:25.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:24 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/4009640281' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:05:26.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:25 smithi086 ceph-mon[25584]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:26.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:25 smithi083 ceph-mon[24738]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:26.240 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:05:25 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:05:25] "GET /metrics HTTP/1.1" 200 214855 "" "Prometheus/2.33.4" 2024-03-04T16:05:26.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:25 smithi083 ceph-mon[29452]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:27.735 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:05:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:05:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:05:28.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:27 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:28.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:27 smithi086 ceph-mon[25584]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:28.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:27 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:28.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:27 smithi083 ceph-mon[24738]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:28.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:27 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:28.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:27 smithi083 ceph-mon[29452]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:30.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:29 smithi086 ceph-mon[25584]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:30.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:29 smithi083 ceph-mon[24738]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:30.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:29 smithi083 ceph-mon[29452]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:31.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:31 smithi086 ceph-mon[25584]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:31.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:31 smithi083 ceph-mon[24738]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:31.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:31 smithi083 ceph-mon[29452]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:34.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:33 smithi086 ceph-mon[25584]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:34.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:33 smithi083 ceph-mon[29452]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:34.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:33 smithi083 ceph-mon[24738]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:36.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:35 smithi086 ceph-mon[25584]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:36.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:35 smithi083 ceph-mon[24738]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:36.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:35 smithi083 ceph-mon[29452]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:36.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:05:35 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:05:35] "GET /metrics HTTP/1.1" 200 214828 "" "Prometheus/2.33.4" 2024-03-04T16:05:37.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:05:37 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:05:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:05:38.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:37 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:38.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:37 smithi086 ceph-mon[25584]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:38.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:37 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:38.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:37 smithi083 ceph-mon[29452]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:38.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:37 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:38.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:37 smithi083 ceph-mon[24738]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:40.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:39 smithi086 ceph-mon[25584]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:40.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:39 smithi083 ceph-mon[24738]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:40.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:39 smithi083 ceph-mon[29452]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:41.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:41 smithi086 ceph-mon[25584]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:41.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:41 smithi083 ceph-mon[24738]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:41.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:41 smithi083 ceph-mon[29452]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:44.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:43 smithi086 ceph-mon[25584]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:44.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:43 smithi083 ceph-mon[24738]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:44.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:43 smithi083 ceph-mon[29452]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:46.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:45 smithi086 ceph-mon[25584]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:46.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:45 smithi083 ceph-mon[24738]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:46.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:05:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:05:45] "GET /metrics HTTP/1.1" 200 214828 "" "Prometheus/2.33.4" 2024-03-04T16:05:46.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:45 smithi083 ceph-mon[29452]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:47.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:05:47 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:05:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:05:48.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:47 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:48.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:47 smithi086 ceph-mon[25584]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:48.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:47 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:48.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:47 smithi083 ceph-mon[29452]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:48.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:47 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:48.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:47 smithi083 ceph-mon[24738]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:50.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:49 smithi086 ceph-mon[25584]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:50.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:49 smithi083 ceph-mon[24738]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:50.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:49 smithi083 ceph-mon[29452]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:51.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:51 smithi086 ceph-mon[25584]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:51.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:51 smithi083 ceph-mon[24738]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:51.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:51 smithi083 ceph-mon[29452]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:53 smithi086 ceph-mon[25584]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:53 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:05:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:53 smithi086 ceph-mon[25584]: Upgrade: Target is version 19.0.0-1826-g9ca0b5de (unknown) 2024-03-04T16:05:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:53 smithi086 ceph-mon[25584]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, digests ['quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977'] 2024-03-04T16:05:54.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:53 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:05:54.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:53 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:05:54.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:53 smithi086 ceph-mon[25584]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-04T16:05:54.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:53 smithi083 ceph-mon[29452]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:54.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:53 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:05:54.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:53 smithi083 ceph-mon[29452]: Upgrade: Target is version 19.0.0-1826-g9ca0b5de (unknown) 2024-03-04T16:05:54.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:53 smithi083 ceph-mon[29452]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, digests ['quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977'] 2024-03-04T16:05:54.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:53 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:05:54.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:53 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:05:54.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:53 smithi083 ceph-mon[29452]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-04T16:05:54.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:53 smithi083 ceph-mon[24738]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:54.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:53 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:05:54.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:53 smithi083 ceph-mon[24738]: Upgrade: Target is version 19.0.0-1826-g9ca0b5de (unknown) 2024-03-04T16:05:54.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:53 smithi083 ceph-mon[24738]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, digests ['quay.ceph.io/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977'] 2024-03-04T16:05:54.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:53 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:05:54.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:53 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:05:54.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:53 smithi083 ceph-mon[24738]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-04T16:05:54.519 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:05:55.099 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:05:55.099 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (3m) 2m ago 3m 15.5M - ba2b418f427c 8a58c6dd3c10 2024-03-04T16:05:55.100 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (3m) 2m ago 3m 46.0M - 8.3.5 dad864ee21e9 9c0f7c6afe4b 2024-03-04T16:05:55.100 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 running (2m) 2m ago 2m 68.0M - 3.5 e1d6a67b021e 553466e95b30 2024-03-04T16:05:55.100 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443 running (8m) 2m ago 8m 411M - 17.2.0 e1d6a67b021e 799a7f926005 2024-03-04T16:05:55.100 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:9283 running (9m) 2m ago 9m 456M - 17.2.0 e1d6a67b021e 5af09a7ff74e 2024-03-04T16:05:55.100 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (9m) 2m ago 9m 46.8M 2048M 17.2.0 e1d6a67b021e d2ba878ed364 2024-03-04T16:05:55.100 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (8m) 2m ago 8m 38.9M 2048M 17.2.0 e1d6a67b021e d1b972ebdd2a 2024-03-04T16:05:55.100 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (8m) 2m ago 8m 38.5M 2048M 17.2.0 e1d6a67b021e e74b0ab73a80 2024-03-04T16:05:55.100 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (4m) 2m ago 4m 14.2M - 1dbe0e931976 7eb96ca65b53 2024-03-04T16:05:55.100 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (4m) 2m ago 4m 14.4M - 1dbe0e931976 9091c36c27a4 2024-03-04T16:05:55.100 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (7m) 2m ago 7m 49.3M 4016M 17.2.0 e1d6a67b021e 56679a9f787b 2024-03-04T16:05:55.100 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (7m) 2m ago 7m 52.2M 4016M 17.2.0 e1d6a67b021e c1070c1a315a 2024-03-04T16:05:55.100 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (7m) 2m ago 7m 45.2M 4016M 17.2.0 e1d6a67b021e 2b913c19f581 2024-03-04T16:05:55.100 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (6m) 2m ago 6m 47.1M 4016M 17.2.0 e1d6a67b021e 9f9a4e2ec5da 2024-03-04T16:05:55.101 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (6m) 2m ago 6m 48.2M 4272M 17.2.0 e1d6a67b021e b3a97f83f1c7 2024-03-04T16:05:55.101 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (6m) 2m ago 5m 45.7M 4272M 17.2.0 e1d6a67b021e b82ca42b7353 2024-03-04T16:05:55.101 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (5m) 2m ago 5m 45.0M 4272M 17.2.0 e1d6a67b021e 18cedf14321d 2024-03-04T16:05:55.101 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (5m) 2m ago 5m 49.7M 4272M 17.2.0 e1d6a67b021e 390e1798c738 2024-03-04T16:05:55.101 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 running (3m) 2m ago 4m 43.2M - 514e6a882f6e 82e50b4a8b86 2024-03-04T16:05:55.101 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (2m) 2m ago 2m 70.5M - 17.2.0 e1d6a67b021e 65f14042f354 2024-03-04T16:05:55.101 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (2m) 2m ago 2m 71.3M - 17.2.0 e1d6a67b021e 78175cc6dc5d 2024-03-04T16:05:55.101 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (2m) 2m ago 2m 68.8M - 17.2.0 e1d6a67b021e 01dcf858bd62 2024-03-04T16:05:55.101 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (2m) 2m ago 2m 73.4M - 17.2.0 e1d6a67b021e 8f70a5ce9f0d 2024-03-04T16:05:55.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:54 smithi086 ceph-mon[25584]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 on smithi086 2024-03-04T16:05:55.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:54 smithi083 ceph-mon[29452]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 on smithi086 2024-03-04T16:05:55.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:54 smithi083 ceph-mon[24738]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 on smithi086 2024-03-04T16:05:55.453 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:05:55.453 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:05:55.453 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-04T16:05:55.453 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:05:55.454 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:05:55.454 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-04T16:05:55.454 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:05:55.454 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:05:55.454 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-04T16:05:55.454 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:05:55.454 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": {}, 2024-03-04T16:05:55.454 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:05:55.454 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-04T16:05:55.454 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:05:55.454 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:05:55.454 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-03-04T16:05:55.454 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:05:55.455 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:05:55.752 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:05:55.752 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:05:55.752 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:05:55.752 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [], 2024-03-04T16:05:55.752 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "0/23 daemons upgraded", 2024-03-04T16:05:55.752 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 image on host smithi086" 2024-03-04T16:05:55.752 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:05:56.099 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_OK 2024-03-04T16:05:56.099 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:05:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:05:55] "GET /metrics HTTP/1.1" 200 214794 "" "Prometheus/2.33.4" 2024-03-04T16:05:56.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:55 smithi083 ceph-mon[24738]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:56.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:55 smithi083 ceph-mon[24738]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:56.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:55 smithi083 ceph-mon[24738]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:56.100 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:55 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3546688089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:05:56.100 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:55 smithi083 ceph-mon[29452]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:56.100 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:55 smithi083 ceph-mon[29452]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:56.100 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:55 smithi083 ceph-mon[29452]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:56.100 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:55 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3546688089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:05:56.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:55 smithi086 ceph-mon[25584]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:56.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:55 smithi086 ceph-mon[25584]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:56.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:55 smithi086 ceph-mon[25584]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:05:56.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:55 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3546688089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:05:57.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:56 smithi086 ceph-mon[25584]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:57.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:56 smithi086 ceph-mon[25584]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:57.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:56 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3537342043' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:05:57.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:56 smithi083 ceph-mon[24738]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:57.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:56 smithi083 ceph-mon[24738]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:57.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:56 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3537342043' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:05:57.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:56 smithi083 ceph-mon[29452]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:57.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:56 smithi083 ceph-mon[29452]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:05:57.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:56 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3537342043' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:05:57.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:05:57 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:05:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:05:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:57 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:57 smithi086 ceph-mon[25584]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:58.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:57 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:58.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:57 smithi083 ceph-mon[24738]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:05:58.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:57 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:58.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:57 smithi083 ceph-mon[29452]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:00.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:05:59 smithi086 ceph-mon[25584]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:00.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:05:59 smithi083 ceph-mon[29452]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:00.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:05:59 smithi083 ceph-mon[24738]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:01.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:01 smithi086 ceph-mon[25584]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:01.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:01 smithi083 ceph-mon[29452]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:01.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:01 smithi083 ceph-mon[24738]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:02 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:06:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:02 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:06:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:06:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:02 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:06:03.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:02 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:06:03.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:02 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:06:03.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:06:03.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:02 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:06:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:02 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:06:03.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:02 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:06:03.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:06:03.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:02 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:06:03.951 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:03 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:06:03.507Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-03-04T16:06:03.951 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:03 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:06:03.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.83:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.83 because it doesn't contain any IP SANs" 2024-03-04T16:06:04.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:03 smithi086 ceph-mon[25584]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:04.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:03 smithi083 ceph-mon[24738]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:04.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:03 smithi083 ceph-mon[29452]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:06.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:05 smithi086 ceph-mon[25584]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:06.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:05 smithi083 ceph-mon[24738]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:06.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:06:05] "GET /metrics HTTP/1.1" 200 214791 "" "Prometheus/2.33.4" 2024-03-04T16:06:06.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:05 smithi083 ceph-mon[29452]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:07.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:07 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:06:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:06:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:07 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:07 smithi086 ceph-mon[25584]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-04T16:06:08.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:07 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:08.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:07 smithi083 ceph-mon[24738]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-04T16:06:08.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:07 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:08.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:07 smithi083 ceph-mon[29452]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-04T16:06:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:09 smithi086 ceph-mon[25584]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:10.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:09 smithi083 ceph-mon[24738]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:10.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:09 smithi083 ceph-mon[29452]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:11.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:11 smithi086 ceph-mon[25584]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-04T16:06:11.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:11 smithi083 ceph-mon[29452]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-04T16:06:11.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:11 smithi083 ceph-mon[24738]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-04T16:06:13.930 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:13 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=error ts=2024-03-04T16:06:13.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.83:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.83 because it doesn't contain any IP SANs" 2024-03-04T16:06:13.930 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:13 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:06:13.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-03-04T16:06:13.930 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:13 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:06:13.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.83:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.83 because it doesn't contain any IP SANs" 2024-03-04T16:06:14.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:13 smithi086 ceph-mon[25584]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-04T16:06:14.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:13 smithi083 ceph-mon[29452]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-04T16:06:14.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:13 smithi083 ceph-mon[24738]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-04T16:06:16.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:15 smithi086 ceph-mon[25584]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:16.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:15 smithi083 ceph-mon[29452]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:16.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:15 smithi083 ceph-mon[24738]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:16.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:15 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:06:15] "GET /metrics HTTP/1.1" 200 214791 "" "Prometheus/2.33.4" 2024-03-04T16:06:17.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:17 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:06:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:06:18.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:17 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:18.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:17 smithi086 ceph-mon[25584]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:18.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:17 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:18.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:17 smithi083 ceph-mon[29452]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:18.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:17 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:18.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:17 smithi083 ceph-mon[24738]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:20.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:19 smithi086 ceph-mon[25584]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:20.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:19 smithi083 ceph-mon[29452]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:20.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:19 smithi083 ceph-mon[24738]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:21.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:21 smithi086 ceph-mon[25584]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:21.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:21 smithi083 ceph-mon[29452]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:21.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:21 smithi083 ceph-mon[24738]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:23.933 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=error ts=2024-03-04T16:06:23.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.83:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.83 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-03-04T16:06:23.933 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:06:23.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.83:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.83 because it doesn't contain any IP SANs" 2024-03-04T16:06:23.933 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:06:23.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-03-04T16:06:24.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:23 smithi086 ceph-mon[25584]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:24.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:23 smithi083 ceph-mon[29452]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:24.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:23 smithi083 ceph-mon[24738]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:26.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:25 smithi086 ceph-mon[25584]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:26.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:25 smithi083 ceph-mon[29452]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:26.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:25 smithi083 ceph-mon[24738]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:26.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:25 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:06:25] "GET /metrics HTTP/1.1" 200 214790 "" "Prometheus/2.33.4" 2024-03-04T16:06:26.426 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:06:27.012 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:06:27.012 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (3m) 3m ago 4m 15.5M - ba2b418f427c 8a58c6dd3c10 2024-03-04T16:06:27.012 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (3m) 3m ago 3m 46.0M - 8.3.5 dad864ee21e9 9c0f7c6afe4b 2024-03-04T16:06:27.013 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 running (3m) 3m ago 3m 68.0M - 3.5 e1d6a67b021e 553466e95b30 2024-03-04T16:06:27.013 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443 running (8m) 3m ago 8m 411M - 17.2.0 e1d6a67b021e 799a7f926005 2024-03-04T16:06:27.013 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:9283 running (10m) 3m ago 10m 456M - 17.2.0 e1d6a67b021e 5af09a7ff74e 2024-03-04T16:06:27.013 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (10m) 3m ago 10m 46.8M 2048M 17.2.0 e1d6a67b021e d2ba878ed364 2024-03-04T16:06:27.013 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (9m) 3m ago 9m 38.9M 2048M 17.2.0 e1d6a67b021e d1b972ebdd2a 2024-03-04T16:06:27.013 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (9m) 3m ago 9m 38.5M 2048M 17.2.0 e1d6a67b021e e74b0ab73a80 2024-03-04T16:06:27.013 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (5m) 3m ago 5m 14.2M - 1dbe0e931976 7eb96ca65b53 2024-03-04T16:06:27.013 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (4m) 3m ago 4m 14.4M - 1dbe0e931976 9091c36c27a4 2024-03-04T16:06:27.013 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (8m) 3m ago 8m 49.3M 4016M 17.2.0 e1d6a67b021e 56679a9f787b 2024-03-04T16:06:27.013 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (8m) 3m ago 8m 52.2M 4016M 17.2.0 e1d6a67b021e c1070c1a315a 2024-03-04T16:06:27.013 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (7m) 3m ago 7m 45.2M 4016M 17.2.0 e1d6a67b021e 2b913c19f581 2024-03-04T16:06:27.013 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (7m) 3m ago 7m 47.1M 4016M 17.2.0 e1d6a67b021e 9f9a4e2ec5da 2024-03-04T16:06:27.013 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (6m) 3m ago 6m 48.2M 4272M 17.2.0 e1d6a67b021e b3a97f83f1c7 2024-03-04T16:06:27.013 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (6m) 3m ago 6m 45.7M 4272M 17.2.0 e1d6a67b021e b82ca42b7353 2024-03-04T16:06:27.014 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (6m) 3m ago 6m 45.0M 4272M 17.2.0 e1d6a67b021e 18cedf14321d 2024-03-04T16:06:27.014 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (5m) 3m ago 5m 49.7M 4272M 17.2.0 e1d6a67b021e 390e1798c738 2024-03-04T16:06:27.014 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 running (3m) 3m ago 5m 43.2M - 514e6a882f6e 82e50b4a8b86 2024-03-04T16:06:27.014 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (3m) 3m ago 3m 70.5M - 17.2.0 e1d6a67b021e 65f14042f354 2024-03-04T16:06:27.014 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (3m) 3m ago 3m 71.3M - 17.2.0 e1d6a67b021e 78175cc6dc5d 2024-03-04T16:06:27.014 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (3m) 3m ago 3m 68.8M - 17.2.0 e1d6a67b021e 01dcf858bd62 2024-03-04T16:06:27.014 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (3m) 3m ago 3m 73.4M - 17.2.0 e1d6a67b021e 8f70a5ce9f0d 2024-03-04T16:06:27.360 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:06:27.360 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:06:27.360 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-04T16:06:27.360 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:06:27.360 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:06:27.360 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-04T16:06:27.360 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:06:27.360 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:06:27.360 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-04T16:06:27.360 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:06:27.360 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": {}, 2024-03-04T16:06:27.361 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:06:27.361 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-04T16:06:27.361 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:06:27.361 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:06:27.361 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-03-04T16:06:27.361 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:06:27.361 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:06:27.656 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:06:27.656 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:06:27.656 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:06:27.656 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [], 2024-03-04T16:06:27.656 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "0/23 daemons upgraded", 2024-03-04T16:06:27.656 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 image on host smithi086" 2024-03-04T16:06:27.657 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:06:27.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[26677]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:06:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:06:28.002 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_OK 2024-03-04T16:06:28.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:27 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:28.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:27 smithi086 ceph-mon[25584]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:28.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:27 smithi086 ceph-mon[25584]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:28.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:27 smithi086 ceph-mon[25584]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:28.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:27 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/2863650304' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:06:28.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:27 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:28.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:27 smithi083 ceph-mon[24738]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:28.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:27 smithi083 ceph-mon[24738]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:28.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:27 smithi083 ceph-mon[24738]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:28.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:27 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2863650304' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:06:28.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:27 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:28.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:27 smithi083 ceph-mon[29452]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:28.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:27 smithi083 ceph-mon[29452]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:28.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:27 smithi083 ceph-mon[29452]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:28.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:27 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/2863650304' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:06:29.195 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:28 smithi086 ceph-mon[25584]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:29.195 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:28 smithi086 ceph-mon[25584]: from='client.25045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:29.195 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:28 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1243329225' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:06:29.195 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:28 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:06:29.195 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:28 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:06:29.195 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:28 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:06:29.195 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:28 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:06:29.195 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:28 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:29.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:28 smithi083 ceph-mon[24738]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:29.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:28 smithi083 ceph-mon[24738]: from='client.25045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:29.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:28 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1243329225' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:06:29.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:28 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:06:29.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:28 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:06:29.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:28 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:06:29.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:28 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:06:29.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:28 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:29.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:28 smithi083 ceph-mon[29452]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:29.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:28 smithi083 ceph-mon[29452]: from='client.25045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:29.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:28 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1243329225' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:06:29.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:28 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:06:29.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:28 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:06:29.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:28 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:06:29.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:28 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:06:29.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:28 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:30.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:29 smithi086 ceph-mon[25584]: Upgrade: Updating mgr.x 2024-03-04T16:06:30.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:29 smithi086 ceph-mon[25584]: Deploying daemon mgr.x on smithi086 2024-03-04T16:06:30.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:29 smithi086 ceph-mon[25584]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:30.235 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:30 smithi086 systemd[1]: Stopping Ceph mgr.x for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:06:30.235 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:30 smithi086 bash[46614]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr.x" found: no such container 2024-03-04T16:06:30.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:29 smithi083 ceph-mon[24738]: Upgrade: Updating mgr.x 2024-03-04T16:06:30.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:29 smithi083 ceph-mon[24738]: Deploying daemon mgr.x on smithi086 2024-03-04T16:06:30.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:29 smithi083 ceph-mon[24738]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:30.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:29 smithi083 ceph-mon[29452]: Upgrade: Updating mgr.x 2024-03-04T16:06:30.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:29 smithi083 ceph-mon[29452]: Deploying daemon mgr.x on smithi086 2024-03-04T16:06:30.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:29 smithi083 ceph-mon[29452]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:30.625 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:30 smithi086 podman[46625]: 2024-03-04 16:06:30.268957969 +0000 UTC m=+0.105810638 container died 799a7f92600598f1844b1642991338a35b4c3a4efb32e443c61b625f82a26abf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x, vcs-type=git, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, RELEASE=HEAD, distribution-scope=public, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.openshift.expose-services=, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, io.openshift.tags=base centos centos-stream, name=centos-stream, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-04T16:06:30.984 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:30 smithi086 podman[46625]: 2024-03-04 16:06:30.624991136 +0000 UTC m=+0.461843797 container cleanup 799a7f92600598f1844b1642991338a35b4c3a4efb32e443c61b625f82a26abf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, version=8, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, release=754, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vcs-type=git, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, architecture=x86_64, io.openshift.expose-services=, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-04T16:06:30.984 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:30 smithi086 bash[46625]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x 2024-03-04T16:06:30.985 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:30 smithi086 podman[46639]: 2024-03-04 16:06:30.776144009 +0000 UTC m=+0.502981736 container remove 799a7f92600598f1844b1642991338a35b4c3a4efb32e443c61b625f82a26abf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, version=8, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.openshift.expose-services=, vcs-type=git, name=centos-stream, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., release=754, architecture=x86_64, com.redhat.component=centos-stream-container, RELEASE=HEAD) 2024-03-04T16:06:30.985 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:30 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-04T16:06:30.985 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:30 smithi086 bash[46651]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr.x" found: no such container 2024-03-04T16:06:30.985 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:30 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-04T16:06:30.985 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:30 smithi086 systemd[1]: Stopped Ceph mgr.x for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:06:30.985 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:30 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mgr.x.service: Consumed 16.619s CPU time. 2024-03-04T16:06:30.988 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:30 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:06:30.558Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-03-04T16:06:31.390 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:31 smithi086 ceph-mon[25584]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:31.390 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:31 smithi086 systemd[1]: Starting Ceph mgr.x for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:06:31.390 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:31 smithi086 podman[46760]: 2024-03-04 16:06:31.161231488 +0000 UTC m=+0.017736224 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:06:31.390 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:31 smithi086 podman[46760]: 2024-03-04 16:06:31.269223971 +0000 UTC m=+0.125728738 container create a73348a7968ee1fa7b096bc4a81a6e44f3ccd788938c8b8de2aee68b540cc45e (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=) 2024-03-04T16:06:31.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:31 smithi083 ceph-mon[24738]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:31.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:31 smithi083 ceph-mon[29452]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:31.642 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:31 smithi086 podman[46760]: 2024-03-04 16:06:31.48830101 +0000 UTC m=+0.344805747 container init a73348a7968ee1fa7b096bc4a81a6e44f3ccd788938c8b8de2aee68b540cc45e (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-04T16:06:31.642 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:31 smithi086 podman[46760]: 2024-03-04 16:06:31.490747932 +0000 UTC m=+0.347252659 container start a73348a7968ee1fa7b096bc4a81a6e44f3ccd788938c8b8de2aee68b540cc45e (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-04T16:06:31.642 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:31 smithi086 ceph-mgr[46779]: -- 172.21.15.86:0/580780803 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b0706405a0 con 0x55b070614c00 2024-03-04T16:06:31.642 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:31 smithi086 bash[46760]: a73348a7968ee1fa7b096bc4a81a6e44f3ccd788938c8b8de2aee68b540cc45e 2024-03-04T16:06:31.642 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:31 smithi086 systemd[1]: Started Ceph mgr.x for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:06:32.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:32 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:32.067+0000 7fc7c9b4e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T16:06:32.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:32 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:32.162+0000 7fc7c9b4e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T16:06:32.932 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:32 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:06:32.932 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:32 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:32.932 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:32 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:06:32.933 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:32 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:32.831+0000 7fc7c9b4e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T16:06:32.933 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:32 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:32.931+0000 7fc7c9b4e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T16:06:32.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:32 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:06:32.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:32 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:32.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:32 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:06:32.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:32 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:06:32.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:32 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:32.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:32 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:06:33.234 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:33.075+0000 7fc7c9b4e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T16:06:33.234 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:33.171+0000 7fc7c9b4e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T16:06:33.553 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:33.256+0000 7fc7c9b4e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T16:06:33.553 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:33.552+0000 7fc7c9b4e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T16:06:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:33 smithi086 ceph-mon[25584]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:33.838 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:33.663+0000 7fc7c9b4e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T16:06:33.838 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:33.759+0000 7fc7c9b4e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T16:06:33.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:33 smithi083 ceph-mon[29452]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:33.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:33 smithi083 ceph-mon[24738]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:33.989 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:33 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=error ts=2024-03-04T16:06:33.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.86:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.83:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.83 because it doesn't contain any IP SANs" 2024-03-04T16:06:33.989 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:33 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:06:33.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-03-04T16:06:33.989 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:33 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:06:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.83:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.83 because it doesn't contain any IP SANs" 2024-03-04T16:06:34.126 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:33.859+0000 7fc7c9b4e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T16:06:34.126 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:34.045+0000 7fc7c9b4e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T16:06:35.203 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:35 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:35.068+0000 7fc7c9b4e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T16:06:35.203 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:35 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:35.202+0000 7fc7c9b4e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T16:06:35.860 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:35 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:35.545+0000 7fc7c9b4e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T16:06:35.860 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:35 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:35.641+0000 7fc7c9b4e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T16:06:35.860 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:35 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:35.858+0000 7fc7c9b4e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:06:36.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:35 smithi086 ceph-mon[25584]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:36.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:35 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:06:36.234 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:36 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:36.061+0000 7fc7c9b4e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T16:06:36.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:35 smithi083 ceph-mon[29452]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:36.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:35 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:06:36.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:35 smithi083 ceph-mon[24738]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:36.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:35 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:06:36.239 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:35 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:06:35] "GET /metrics HTTP/1.1" 200 214792 "" "Prometheus/2.33.4" 2024-03-04T16:06:36.655 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:36 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:36.365+0000 7fc7c9b4e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:06:36.655 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:36 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:36.654+0000 7fc7c9b4e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T16:06:36.984 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:36 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:36.898+0000 7fc7c9b4e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T16:06:37.704 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:37 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:06:37.704 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:37 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:37.704 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:37 smithi086 ceph-mon[25584]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:37.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:37 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:06:37.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:37 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:37.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:37 smithi083 ceph-mon[24738]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:37.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:37 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:06:37.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:37 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:37.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:37 smithi083 ceph-mon[29452]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:06:37.957 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:37 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:37.721+0000 7fc7c9b4e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T16:06:38.645 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:38 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:38.314+0000 7fc7c9b4e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T16:06:38.645 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:38 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:38.417+0000 7fc7c9b4e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T16:06:38.645 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:38 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:38.644+0000 7fc7c9b4e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T16:06:38.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:38 smithi086 ceph-mon[25584]: Standby manager daemon x restarted 2024-03-04T16:06:38.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:38 smithi086 ceph-mon[25584]: Standby manager daemon x started 2024-03-04T16:06:38.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:38 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T16:06:38.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:38 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:06:38.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:38 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T16:06:38.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:38 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:06:38.985 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:38 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:06:38] ENGINE Bus STARTING 2024-03-04T16:06:38.985 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:38 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: CherryPy Checker: 2024-03-04T16:06:38.985 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:38 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: The Application mounted at '' has an empty config. 2024-03-04T16:06:38.985 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:38 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:06:38.985 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:38 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:06:38] ENGINE Serving on http://:::9283 2024-03-04T16:06:38.985 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:38 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:06:38] ENGINE Bus STARTED 2024-03-04T16:06:38.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:38 smithi083 ceph-mon[24738]: Standby manager daemon x restarted 2024-03-04T16:06:38.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:38 smithi083 ceph-mon[24738]: Standby manager daemon x started 2024-03-04T16:06:38.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:38 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T16:06:38.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:38 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:06:38.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:38 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T16:06:38.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:38 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:06:38.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:38 smithi083 ceph-mon[29452]: Standby manager daemon x restarted 2024-03-04T16:06:38.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:38 smithi083 ceph-mon[29452]: Standby manager daemon x started 2024-03-04T16:06:38.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:38 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T16:06:38.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:38 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:06:38.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:38 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T16:06:38.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:38 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:06:39.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:39 smithi086 ceph-mon[25584]: mgrmap e21: y(active, since 5m), standbys: x 2024-03-04T16:06:39.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:39 smithi086 ceph-mon[25584]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:39.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:39 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:06:39.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:39 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:06:39.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:39 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:06:39.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:39 smithi086 ceph-mon[25584]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-03-04T16:06:39.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:39 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-03-04T16:06:39.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:39 smithi086 ceph-mon[25584]: osdmap e97: 8 total, 8 up, 8 in 2024-03-04T16:06:39.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:39 smithi083 ceph-mon[24738]: mgrmap e21: y(active, since 5m), standbys: x 2024-03-04T16:06:39.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:39 smithi083 ceph-mon[24738]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:39.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:39 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:06:39.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:39 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:06:39.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:39 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:06:39.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:39 smithi083 ceph-mon[24738]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-03-04T16:06:39.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:39 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-03-04T16:06:39.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:39 smithi083 ceph-mon[24738]: osdmap e97: 8 total, 8 up, 8 in 2024-03-04T16:06:39.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:39 smithi083 ceph-mon[29452]: mgrmap e21: y(active, since 5m), standbys: x 2024-03-04T16:06:39.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:39 smithi083 ceph-mon[29452]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:06:39.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:39 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:06:39.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:39 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' 2024-03-04T16:06:39.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:39 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:06:39.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:39 smithi083 ceph-mon[29452]: from='mgr.24337 172.21.15.83:0/2269800059' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-03-04T16:06:39.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:39 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-03-04T16:06:39.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:39 smithi083 ceph-mon[29452]: osdmap e97: 8 total, 8 up, 8 in 2024-03-04T16:06:40.733 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:40.398+0000 7fa66a33b700 -1 mgr handle_mgr_map I was active but no longer amignoring --setuser ceph since I am not root 2024-03-04T16:06:40.734 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ignoring --setgroup ceph since I am not root 2024-03-04T16:06:40.734 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mgr[24974]: -- 172.21.15.83:0/3162677628 <== mon.1 v2:172.21.15.83:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5588e852c340 con 0x5588e8505000 2024-03-04T16:06:40.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:06:40] ENGINE Bus STOPPING 2024-03-04T16:06:40.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:06:40] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-04T16:06:40.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:06:40] ENGINE Bus STOPPED 2024-03-04T16:06:40.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:06:40] ENGINE Bus STARTING 2024-03-04T16:06:40.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:06:40] ENGINE Serving on http://:::9283 2024-03-04T16:06:40.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:06:40] ENGINE Bus STARTED 2024-03-04T16:06:40.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-04T16:06:40.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: Failing over to other MGR 2024-03-04T16:06:40.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-03-04T16:06:40.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: mgrmap e22: x(active, starting, since 0.9945s) 2024-03-04T16:06:40.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:06:40.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:06:40.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:06:40.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T16:06:40.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:06:40.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:06:40.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:06:40.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:06:40.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:06:40.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:06:40.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:06:40.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:06:40.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:06:40.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:06:40.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:06:40.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: Manager daemon x is now available 2024-03-04T16:06:40.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:40.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: Queued rgw.foo for migration 2024-03-04T16:06:40.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:40.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: Queued rgw.smpl for migration 2024-03-04T16:06:40.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-04T16:06:40.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-03-04T16:06:40.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:40.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:06:40.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:40.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:06:40.987 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:06:40.987 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-04T16:06:40.987 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:40 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-04T16:06:40.987 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:40.929+0000 7f0eb9a40000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T16:06:40.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-04T16:06:40.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: Failing over to other MGR 2024-03-04T16:06:40.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-03-04T16:06:40.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: mgrmap e22: x(active, starting, since 0.9945s) 2024-03-04T16:06:40.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:06:40.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:06:40.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:06:40.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T16:06:40.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:06:40.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:06:40.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:06:40.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:06:40.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:06:40.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:06:40.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:06:40.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:06:40.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:06:40.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:06:40.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:06:40.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: Manager daemon x is now available 2024-03-04T16:06:40.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:40.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: Queued rgw.foo for migration 2024-03-04T16:06:40.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:40.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: Queued rgw.smpl for migration 2024-03-04T16:06:40.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-04T16:06:40.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-03-04T16:06:40.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:40.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:06:40.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:40.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:06:40.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:06:40.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-04T16:06:40.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-04T16:06:40.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-04T16:06:40.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: Failing over to other MGR 2024-03-04T16:06:40.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.24337 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-03-04T16:06:40.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: mgrmap e22: x(active, starting, since 0.9945s) 2024-03-04T16:06:40.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:06:40.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:06:40.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:06:40.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T16:06:40.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:06:40.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:06:40.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:06:40.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:06:40.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:06:40.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:06:40.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:06:40.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:06:40.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:06:40.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:06:40.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:06:40.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: Manager daemon x is now available 2024-03-04T16:06:40.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:40.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: Queued rgw.foo for migration 2024-03-04T16:06:40.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:40.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: Queued rgw.smpl for migration 2024-03-04T16:06:40.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-04T16:06:40.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-03-04T16:06:40.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:40.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:06:40.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:40.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:06:40.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:06:40.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-04T16:06:40.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:40 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-04T16:06:41.488 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:41 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:41.026+0000 7f0eb9a40000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T16:06:41.488 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:41 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:06:41.223Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-03-04T16:06:41.934 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:41 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:41.641+0000 7f0eb9a40000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T16:06:41.934 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:41 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:41.731+0000 7f0eb9a40000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T16:06:41.934 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:41 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:41.816+0000 7f0eb9a40000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T16:06:41.935 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:41 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:41.935 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:41 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:41.935 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:41 smithi083 ceph-mon[29452]: Deploying cephadm binary to smithi083 2024-03-04T16:06:41.935 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:41 smithi083 ceph-mon[29452]: Deploying cephadm binary to smithi086 2024-03-04T16:06:41.935 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:41 smithi083 ceph-mon[29452]: mgrmap e23: x(active, since 1.99796s) 2024-03-04T16:06:41.935 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:41 smithi083 ceph-mon[29452]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:06:41.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:41 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:41.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:41 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:41.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:41 smithi083 ceph-mon[24738]: Deploying cephadm binary to smithi083 2024-03-04T16:06:41.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:41 smithi083 ceph-mon[24738]: Deploying cephadm binary to smithi086 2024-03-04T16:06:41.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:41 smithi083 ceph-mon[24738]: mgrmap e23: x(active, since 1.99796s) 2024-03-04T16:06:41.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:41 smithi083 ceph-mon[24738]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:06:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:41 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:41 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:41 smithi086 ceph-mon[25584]: Deploying cephadm binary to smithi083 2024-03-04T16:06:42.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:41 smithi086 ceph-mon[25584]: Deploying cephadm binary to smithi086 2024-03-04T16:06:42.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:41 smithi086 ceph-mon[25584]: mgrmap e23: x(active, since 1.99796s) 2024-03-04T16:06:42.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:41 smithi086 ceph-mon[25584]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:06:42.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:41 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:41.971+0000 7f0eb9a40000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T16:06:42.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:42 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:42.051+0000 7f0eb9a40000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T16:06:42.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:42 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:42.160+0000 7f0eb9a40000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T16:06:42.497 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:42 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:42.244+0000 7f0eb9a40000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T16:06:42.497 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:42 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:42.410+0000 7f0eb9a40000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T16:06:42.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:42 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:42.496+0000 7f0eb9a40000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T16:06:43.665 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:43 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:43.533+0000 7f0eb9a40000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T16:06:43.665 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:43 smithi083 ceph-mon[24738]: [04/Mar/2024:16:06:41] ENGINE Bus STARTING 2024-03-04T16:06:43.665 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:43 smithi083 ceph-mon[24738]: [04/Mar/2024:16:06:41] ENGINE Serving on http://172.21.15.86:8765 2024-03-04T16:06:43.665 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:43 smithi083 ceph-mon[24738]: [04/Mar/2024:16:06:42] ENGINE Serving on https://172.21.15.86:7150 2024-03-04T16:06:43.665 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:43 smithi083 ceph-mon[24738]: [04/Mar/2024:16:06:42] ENGINE Bus STARTED 2024-03-04T16:06:43.665 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:43 smithi083 ceph-mon[24738]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:06:43.666 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:43 smithi083 ceph-mon[29452]: [04/Mar/2024:16:06:41] ENGINE Bus STARTING 2024-03-04T16:06:43.666 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:43 smithi083 ceph-mon[29452]: [04/Mar/2024:16:06:41] ENGINE Serving on http://172.21.15.86:8765 2024-03-04T16:06:43.666 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:43 smithi083 ceph-mon[29452]: [04/Mar/2024:16:06:42] ENGINE Serving on https://172.21.15.86:7150 2024-03-04T16:06:43.666 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:43 smithi083 ceph-mon[29452]: [04/Mar/2024:16:06:42] ENGINE Bus STARTED 2024-03-04T16:06:43.666 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:43 smithi083 ceph-mon[29452]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:06:43.666 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:43 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:06:43.400Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=7 err="Post \"https://172.21.15.83:8443//api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:06:43.666 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:43 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=error ts=2024-03-04T16:06:43.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.83:8443//api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:06:43.667 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:43 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:06:43.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.83:8443//api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:06:43.667 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:43 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:06:43.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-03-04T16:06:43.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:43 smithi086 ceph-mon[25584]: [04/Mar/2024:16:06:41] ENGINE Bus STARTING 2024-03-04T16:06:43.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:43 smithi086 ceph-mon[25584]: [04/Mar/2024:16:06:41] ENGINE Serving on http://172.21.15.86:8765 2024-03-04T16:06:43.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:43 smithi086 ceph-mon[25584]: [04/Mar/2024:16:06:42] ENGINE Serving on https://172.21.15.86:7150 2024-03-04T16:06:43.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:43 smithi086 ceph-mon[25584]: [04/Mar/2024:16:06:42] ENGINE Bus STARTED 2024-03-04T16:06:43.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:43 smithi086 ceph-mon[25584]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:06:43.948 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:43 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:43.664+0000 7f0eb9a40000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T16:06:43.949 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:43 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:43.859+0000 7f0eb9a40000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T16:06:44.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:43 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:43.947+0000 7f0eb9a40000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T16:06:44.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:44 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:44.085+0000 7f0eb9a40000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:06:44.723 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:44 smithi083 ceph-mon[24738]: mgrmap e24: x(active, since 4s) 2024-03-04T16:06:44.724 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:44 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:44.410+0000 7f0eb9a40000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T16:06:44.724 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:44 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:44.635+0000 7f0eb9a40000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:06:44.724 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:44 smithi083 ceph-mon[29452]: mgrmap e24: x(active, since 4s) 2024-03-04T16:06:44.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:44 smithi086 ceph-mon[25584]: mgrmap e24: x(active, since 4s) 2024-03-04T16:06:44.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:44 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:44.723+0000 7f0eb9a40000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T16:06:44.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:44 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:44.897+0000 7f0eb9a40000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T16:06:45.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:45 smithi086 ceph-mon[25584]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:06:45.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:45 smithi083 ceph-mon[29452]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:06:45.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:45 smithi083 ceph-mon[24738]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:06:46.166 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:45.849+0000 7f0eb9a40000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T16:06:46.166 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:46 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:46.165+0000 7f0eb9a40000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T16:06:46.427 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:46 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:46.266+0000 7f0eb9a40000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T16:06:46.427 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:46 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:46.426+0000 7f0eb9a40000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T16:06:46.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:46 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: [04/Mar/2024:16:06:46] ENGINE Bus STARTING 2024-03-04T16:06:46.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:46 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: CherryPy Checker: 2024-03-04T16:06:46.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:46 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: The Application mounted at '' has an empty config. 2024-03-04T16:06:46.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:46 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: 2024-03-04T16:06:46.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:46 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: [04/Mar/2024:16:06:46] ENGINE Serving on http://:::9283 2024-03-04T16:06:46.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:46 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: [04/Mar/2024:16:06:46] ENGINE Bus STARTED 2024-03-04T16:06:47.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:47 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:47.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:47 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:47.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:47 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:47.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:47 smithi086 ceph-mon[25584]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:06:47.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:47 smithi086 ceph-mon[25584]: Standby manager daemon y started 2024-03-04T16:06:47.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:47 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.83:0/3358156030' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-04T16:06:47.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:47 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.83:0/3358156030' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:06:47.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:47 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.83:0/3358156030' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-04T16:06:47.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:47 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.83:0/3358156030' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:06:47.238 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:46 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:06:46.965Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=5 err="Post \"https://172.21.15.83:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.83 because it doesn't contain any IP SANs" 2024-03-04T16:06:47.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:47 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:47.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:47 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:47.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:47 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:47.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:47 smithi083 ceph-mon[24738]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:06:47.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:47 smithi083 ceph-mon[24738]: Standby manager daemon y started 2024-03-04T16:06:47.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:47 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.83:0/3358156030' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-04T16:06:47.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:47 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.83:0/3358156030' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:06:47.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:47 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.83:0/3358156030' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-04T16:06:47.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:47 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.83:0/3358156030' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:06:47.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:47 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:47.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:47 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:47.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:47 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:47.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:47 smithi083 ceph-mon[29452]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:06:47.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:47 smithi083 ceph-mon[29452]: Standby manager daemon y started 2024-03-04T16:06:47.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:47 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.83:0/3358156030' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-04T16:06:47.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:47 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.83:0/3358156030' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:06:47.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:47 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.83:0/3358156030' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-04T16:06:47.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:47 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.83:0/3358156030' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:06:47.678 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:47 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:06:47] "GET /metrics HTTP/1.1" 200 35141 "" "Prometheus/2.33.4" 2024-03-04T16:06:48.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:48 smithi086 ceph-mon[25584]: mgrmap e25: x(active, since 7s), standbys: y 2024-03-04T16:06:48.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:48 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T16:06:48.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:48 smithi083 ceph-mon[24738]: mgrmap e25: x(active, since 7s), standbys: y 2024-03-04T16:06:48.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:48 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T16:06:48.351 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:48 smithi083 ceph-mon[29452]: mgrmap e25: x(active, since 7s), standbys: y 2024-03-04T16:06:48.352 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:48 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T16:06:49.349 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:49 smithi083 ceph-mon[29452]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-04T16:06:49.350 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:49 smithi083 ceph-mon[24738]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-04T16:06:49.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:49 smithi086 ceph-mon[25584]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-04T16:06:50.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:50 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:50.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:50 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:50.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:50 smithi086 ceph-mon[25584]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-04T16:06:50.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:50 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:50.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:50 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:50.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:50 smithi083 ceph-mon[29452]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-04T16:06:50.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:50 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:50.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:50 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:50.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:50 smithi083 ceph-mon[24738]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-04T16:06:51.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:51 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:51.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:51 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:51.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:51 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:51 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:51 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:51 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:51 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:51 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:51 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:51 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:51 smithi086 ceph-mon[25584]: Adjusting osd_memory_target on smithi083 to 2736M 2024-03-04T16:06:51.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:51 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:51.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:51.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:51.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[24738]: Adjusting osd_memory_target on smithi083 to 2736M 2024-03-04T16:06:51.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:51.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:51.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:51.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:06:51.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[29452]: Adjusting osd_memory_target on smithi083 to 2736M 2024-03-04T16:06:51.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:51 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:52.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:52 smithi086 ceph-mon[25584]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-04T16:06:52.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:52 smithi083 ceph-mon[29452]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-04T16:06:52.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:52 smithi083 ceph-mon[24738]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-04T16:06:53.988 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:53 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=error ts=2024-03-04T16:06:53.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.83:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.83 because it doesn't contain any IP SANs" 2024-03-04T16:06:53.988 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:53 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:06:53.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.83:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.83 because it doesn't contain any IP SANs" 2024-03-04T16:06:53.988 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:06:53 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:06:53.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-03-04T16:06:55.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:55 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:55.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:55 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:55.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:55 smithi086 ceph-mon[25584]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:06:55.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:55 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:55.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:55 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:55.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:55 smithi083 ceph-mon[24738]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:06:55.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:55 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:55.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:55 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:55.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:55 smithi083 ceph-mon[29452]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:06:56.163 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:06:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:06:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:06:56.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:56 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:56.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:56 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:56.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:56 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:56.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:56 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:56.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:56 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:56.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:56 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:56.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:56 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:56.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:56 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:06:56.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:56 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:57.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:57 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:57.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:57 smithi086 ceph-mon[25584]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:06:57.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:06:57 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:06:57] "GET /metrics HTTP/1.1" 200 38140 "" "Prometheus/2.33.4" 2024-03-04T16:06:57.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:57 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:57.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:57 smithi083 ceph-mon[29452]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:06:57.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:57 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:57.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:57 smithi083 ceph-mon[24738]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:06:58.365 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:06:59.005 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:06:59.005 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (4m) 9s ago 4m 26.3M - 0.23.0 ba2b418f427c 8a58c6dd3c10 2024-03-04T16:06:59.005 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (4m) 4s ago 4m 53.8M - 8.3.5 dad864ee21e9 9c0f7c6afe4b 2024-03-04T16:06:59.005 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 running (3m) 9s ago 3m 53.4M - 3.5 e1d6a67b021e 553466e95b30 2024-03-04T16:06:59.005 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443,9283 running (27s) 4s ago 9m 490M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e a73348a7968e 2024-03-04T16:06:59.005 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:9283 running (10m) 9s ago 10m 408M - 17.2.0 e1d6a67b021e 5af09a7ff74e 2024-03-04T16:06:59.005 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (10m) 9s ago 10m 50.2M 2048M 17.2.0 e1d6a67b021e d2ba878ed364 2024-03-04T16:06:59.005 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (9m) 4s ago 9m 39.6M 2048M 17.2.0 e1d6a67b021e d1b972ebdd2a 2024-03-04T16:06:59.006 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (9m) 9s ago 9m 39.7M 2048M 17.2.0 e1d6a67b021e e74b0ab73a80 2024-03-04T16:06:59.006 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (5m) 9s ago 5m 16.9M - 1.3.1 1dbe0e931976 7eb96ca65b53 2024-03-04T16:06:59.006 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (5m) 4s ago 5m 16.6M - 1.3.1 1dbe0e931976 9091c36c27a4 2024-03-04T16:06:59.006 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (9m) 9s ago 8m 51.0M 2736M 17.2.0 e1d6a67b021e 56679a9f787b 2024-03-04T16:06:59.006 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (8m) 9s ago 8m 55.3M 2736M 17.2.0 e1d6a67b021e c1070c1a315a 2024-03-04T16:06:59.006 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (8m) 9s ago 8m 47.2M 2736M 17.2.0 e1d6a67b021e 2b913c19f581 2024-03-04T16:06:59.006 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (7m) 9s ago 7m 49.3M 2736M 17.2.0 e1d6a67b021e 9f9a4e2ec5da 2024-03-04T16:06:59.006 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (7m) 4s ago 7m 51.5M 4272M 17.2.0 e1d6a67b021e b3a97f83f1c7 2024-03-04T16:06:59.006 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (7m) 4s ago 7m 49.6M 4272M 17.2.0 e1d6a67b021e b82ca42b7353 2024-03-04T16:06:59.006 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (6m) 4s ago 6m 48.0M 4272M 17.2.0 e1d6a67b021e 18cedf14321d 2024-03-04T16:06:59.006 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (6m) 4s ago 6m 52.2M 4272M 17.2.0 e1d6a67b021e 390e1798c738 2024-03-04T16:06:59.006 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 running (4m) 4s ago 5m 66.6M - 2.33.4 514e6a882f6e 82e50b4a8b86 2024-03-04T16:06:59.007 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (4m) 9s ago 4m 72.0M - 17.2.0 e1d6a67b021e 65f14042f354 2024-03-04T16:06:59.007 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (4m) 4s ago 4m 73.0M - 17.2.0 e1d6a67b021e 78175cc6dc5d 2024-03-04T16:06:59.007 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (3m) 9s ago 3m 70.2M - 17.2.0 e1d6a67b021e 01dcf858bd62 2024-03-04T16:06:59.007 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (3m) 4s ago 3m 75.0M - 17.2.0 e1d6a67b021e 8f70a5ce9f0d 2024-03-04T16:06:59.391 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:06:59.392 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:06:59.392 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-04T16:06:59.392 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:06:59.392 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:06:59.392 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-04T16:06:59.392 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 1 2024-03-04T16:06:59.393 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:06:59.393 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:06:59.393 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-04T16:06:59.393 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:06:59.393 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": {}, 2024-03-04T16:06:59.393 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:06:59.394 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-04T16:06:59.394 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:06:59.394 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:06:59.394 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-03-04T16:06:59.394 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 1 2024-03-04T16:06:59.394 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:06:59.395 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:06:59.576 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:59 smithi086 ceph-mon[25584]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:59.577 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:59 smithi086 ceph-mon[25584]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:06:59.577 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:06:59 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1885562788' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:06:59.721 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:06:59.721 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:06:59.722 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:06:59.722 INFO:teuthology.orchestra.run.smithi083.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:06:59.722 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [], 2024-03-04T16:06:59.722 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "1/23 daemons upgraded", 2024-03-04T16:06:59.722 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "", 2024-03-04T16:06:59.722 INFO:teuthology.orchestra.run.smithi083.stdout: "is_paused": false 2024-03-04T16:06:59.722 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:06:59.723 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:59 smithi083 ceph-mon[29452]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:59.723 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:59 smithi083 ceph-mon[29452]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:06:59.723 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:06:59 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1885562788' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:06:59.723 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:59 smithi083 ceph-mon[24738]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:06:59.723 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:59 smithi083 ceph-mon[24738]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:06:59.724 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:06:59 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1885562788' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:00.130 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_OK 2024-03-04T16:07:00.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:00 smithi086 ceph-mon[25584]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:00.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:00 smithi086 ceph-mon[25584]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:00.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:00 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:00.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:00 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:00.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:00 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:00 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:00 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:00 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:00 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:00 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:00 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:00 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:00 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:00.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:00 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:00.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:00 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:07:00.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:00 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1758002022' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:07:00.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[24738]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:00.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[24738]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:00.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:00.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:00.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:00.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:00.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:07:00.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1758002022' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:07:00.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[29452]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:00.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[29452]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:00.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:00.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:00.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:07:00.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:00.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:00.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:07:00.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:00 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1758002022' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:07:01.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: Adjusting osd_memory_target on smithi086 to 2992M 2024-03-04T16:07:01.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: Updating smithi083:/etc/ceph/ceph.conf 2024-03-04T16:07:01.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-04T16:07:01.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: from='client.25117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:01.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: Updating smithi086:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:07:01.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: Updating smithi083:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:07:01.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:07:01.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:07:01.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:01.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: Updating smithi083:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:07:01.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: Updating smithi086:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:07:01.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:01 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.738 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:01 smithi083 systemd[1]: Stopping Ceph node-exporter.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:07:01.738 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:01 smithi083 bash[58714]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter.a" found: no such container 2024-03-04T16:07:01.738 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:01 smithi083 podman[58727]: 2024-03-04 16:07:01.496739756 +0000 UTC m=+0.137212868 container died 7eb96ca65b539c2b0794cf3529a7c96782ddaeb04439ec299608bd0aecbc2422 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T16:07:01.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: Adjusting osd_memory_target on smithi086 to 2992M 2024-03-04T16:07:01.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: Updating smithi083:/etc/ceph/ceph.conf 2024-03-04T16:07:01.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-04T16:07:01.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: from='client.25117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:01.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: Updating smithi086:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:07:01.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: Updating smithi083:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:07:01.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:07:01.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:07:01.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:01.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: Updating smithi083:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:07:01.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: Updating smithi086:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:07:01.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: Adjusting osd_memory_target on smithi086 to 2992M 2024-03-04T16:07:01.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: Updating smithi083:/etc/ceph/ceph.conf 2024-03-04T16:07:01.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-04T16:07:01.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: from='client.25117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:01.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: Updating smithi086:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:07:01.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: Updating smithi083:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:07:01.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:07:01.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:07:01.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:01.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: Updating smithi083:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:07:01.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: Updating smithi086:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:07:01.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:01.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:01 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:02.238 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:01 smithi083 podman[58727]: 2024-03-04 16:07:01.944782154 +0000 UTC m=+0.585255253 container cleanup 7eb96ca65b539c2b0794cf3529a7c96782ddaeb04439ec299608bd0aecbc2422 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T16:07:02.238 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:01 smithi083 bash[58727]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a 2024-03-04T16:07:02.238 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:01 smithi083 bash[58754]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter.a" found: no such container 2024-03-04T16:07:02.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-mon[29452]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-04T16:07:02.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-mon[29452]: Reconfiguring daemon node-exporter.a on smithi083 2024-03-04T16:07:02.614 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-mon[24738]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-04T16:07:02.614 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-mon[24738]: Reconfiguring daemon node-exporter.a on smithi083 2024-03-04T16:07:02.614 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 podman[58742]: 2024-03-04 16:07:02.354764694 +0000 UTC m=+0.862644766 container remove 7eb96ca65b539c2b0794cf3529a7c96782ddaeb04439ec299608bd0aecbc2422 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T16:07:02.615 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-04T16:07:02.615 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-04T16:07:02.615 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 systemd[1]: Stopped Ceph node-exporter.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:07:02.615 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@node-exporter.a.service: Consumed 2.693s CPU time. 2024-03-04T16:07:02.615 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 systemd[1]: Starting Ceph node-exporter.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:07:02.615 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 podman[58825]: 2024-03-04 16:07:02.497125287 +0000 UTC m=+0.018112547 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-04T16:07:02.615 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 podman[58825]: 2024-03-04 16:07:02.614231009 +0000 UTC m=+0.135218252 container create 9bf54b46064062273c6ec8a493b9c8ba473508399d146c29305015aca7f748d5 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T16:07:02.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:02 smithi086 ceph-mon[25584]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-04T16:07:02.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:02 smithi086 ceph-mon[25584]: Reconfiguring daemon node-exporter.a on smithi083 2024-03-04T16:07:02.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:02 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:07:02.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.86:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.83:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:07:02.925 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 podman[58825]: 2024-03-04 16:07:02.808209375 +0000 UTC m=+0.329196616 container init 9bf54b46064062273c6ec8a493b9c8ba473508399d146c29305015aca7f748d5 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T16:07:02.925 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 podman[58825]: 2024-03-04 16:07:02.810987967 +0000 UTC m=+0.331975216 container start 9bf54b46064062273c6ec8a493b9c8ba473508399d146c29305015aca7f748d5 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T16:07:02.925 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.814Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-04T16:07:02.925 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.814Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-04T16:07:02.925 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-04T16:07:02.925 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-04T16:07:02.926 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-04T16:07:02.926 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=arp 2024-03-04T16:07:02.926 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-04T16:07:02.926 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-04T16:07:02.926 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-04T16:07:02.926 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-04T16:07:02.926 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-04T16:07:02.926 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-04T16:07:02.926 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-04T16:07:02.926 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-04T16:07:02.926 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=edac 2024-03-04T16:07:02.927 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-04T16:07:02.927 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-04T16:07:02.927 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-04T16:07:02.927 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-04T16:07:02.927 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-04T16:07:02.927 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-04T16:07:02.927 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-04T16:07:02.927 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-04T16:07:02.927 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-04T16:07:02.928 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-04T16:07:02.928 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-04T16:07:02.928 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-04T16:07:02.928 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-04T16:07:02.928 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-04T16:07:02.928 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-04T16:07:02.928 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-04T16:07:02.928 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=os 2024-03-04T16:07:02.928 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-04T16:07:02.928 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-04T16:07:02.928 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-04T16:07:02.928 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-04T16:07:02.928 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-04T16:07:02.929 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-04T16:07:02.929 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=stat 2024-03-04T16:07:02.929 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-04T16:07:02.929 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-04T16:07:02.929 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-04T16:07:02.929 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=time 2024-03-04T16:07:02.929 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-04T16:07:02.929 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=uname 2024-03-04T16:07:02.929 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-04T16:07:02.929 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-04T16:07:02.929 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-04T16:07:02.929 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.815Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-04T16:07:02.929 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a[58840]: ts=2024-03-04T16:07:02.816Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-04T16:07:02.929 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 bash[58825]: 9bf54b46064062273c6ec8a493b9c8ba473508399d146c29305015aca7f748d5 2024-03-04T16:07:02.929 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:07:02 smithi083 systemd[1]: Started Ceph node-exporter.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:07:03.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:03 smithi086 ceph-mon[25584]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:03.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:03 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:03.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:03 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:03.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:03 smithi083 ceph-mon[24738]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:03.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:03 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:03.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:03 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:03.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:03 smithi083 ceph-mon[29452]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:03.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:03 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:03.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:03 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:03.739 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:03 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=error ts=2024-03-04T16:07:03.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.83:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.83 because it doesn't contain any IP SANs" 2024-03-04T16:07:03.739 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:03 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:07:03.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.83:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.83 because it doesn't contain any IP SANs" 2024-03-04T16:07:03.739 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:03 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=warn ts=2024-03-04T16:07:03.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-03-04T16:07:04.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:04 smithi086 ceph-mon[25584]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-04T16:07:04.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:04 smithi086 ceph-mon[25584]: Reconfiguring daemon alertmanager.a on smithi083 2024-03-04T16:07:04.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:04 smithi083 ceph-mon[29452]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-04T16:07:04.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:04 smithi083 ceph-mon[29452]: Reconfiguring daemon alertmanager.a on smithi083 2024-03-04T16:07:04.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:04 smithi083 ceph-mon[24738]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-04T16:07:04.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:04 smithi083 ceph-mon[24738]: Reconfiguring daemon alertmanager.a on smithi083 2024-03-04T16:07:05.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:05 smithi083 ceph-mon[29452]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:05.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:05 smithi083 ceph-mon[24738]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:05.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:05 smithi086 ceph-mon[25584]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:06.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:07:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:07:07.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:07 smithi086 ceph-mon[25584]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:07.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:07 smithi086 ceph-mon[25584]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:07.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:07 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:07:07] "GET /metrics HTTP/1.1" 200 38140 "" "Prometheus/2.33.4" 2024-03-04T16:07:07.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:07 smithi083 ceph-mon[29452]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:07.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:07 smithi083 ceph-mon[29452]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:07.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:07 smithi083 ceph-mon[24738]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:07.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:07 smithi083 ceph-mon[24738]: from='client.34688 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:09.903 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:09 smithi083 ceph-mon[24738]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:09.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:09 smithi083 ceph-mon[29452]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:09.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:09 smithi086 ceph-mon[25584]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:11.415 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:11 smithi083 ceph-mon[29452]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:11.415 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:11 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:11.416 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:11 smithi083 ceph-mon[24738]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:11.416 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:11 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:11.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:11 smithi086 ceph-mon[25584]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:11.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:11 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:12.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:12 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:07:12.315Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.86:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.83:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:07:12.857 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:12 smithi083 systemd[1]: Stopping Ceph alertmanager.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:07:12.857 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:12 smithi083 bash[59102]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager.a" found: no such container 2024-03-04T16:07:13.175 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:12 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[49515]: level=info ts=2024-03-04T16:07:12.856Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-04T16:07:13.175 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:12 smithi083 podman[59115]: 2024-03-04 16:07:12.877377631 +0000 UTC m=+0.132195739 container died 8a58c6dd3c10921b2be122535071431d8504a354f17b43abb36923dbd191dde3 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:07:13.465 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:13 smithi083 podman[59115]: 2024-03-04 16:07:13.281773237 +0000 UTC m=+0.536591342 container cleanup 8a58c6dd3c10921b2be122535071431d8504a354f17b43abb36923dbd191dde3 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:07:13.466 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:13 smithi083 bash[59115]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a 2024-03-04T16:07:13.466 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:13 smithi083 bash[59144]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager.a" found: no such container 2024-03-04T16:07:13.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:13 smithi086 ceph-mon[25584]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:13.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:13 smithi083 ceph-mon[24738]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:13.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:13 smithi083 ceph-mon[29452]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:13.738 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:13 smithi083 podman[59130]: 2024-03-04 16:07:13.46601716 +0000 UTC m=+0.588257034 container remove 8a58c6dd3c10921b2be122535071431d8504a354f17b43abb36923dbd191dde3 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:07:13.739 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:13 smithi083 podman[59130]: 2024-03-04 16:07:13.575569354 +0000 UTC m=+0.697809227 volume remove 2ac12bd63644fc0ccd25c86394196b2422cce79643cf06b737b8c58e696f3333 2024-03-04T16:07:13.739 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:13 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-04T16:07:13.739 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:13 smithi083 systemd[1]: Stopped Ceph alertmanager.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:07:13.739 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:13 smithi083 systemd[1]: Starting Ceph alertmanager.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:07:14.127 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:13 smithi083 podman[59210]: 2024-03-04 16:07:13.715151183 +0000 UTC m=+0.017648994 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-04T16:07:14.128 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:13 smithi083 podman[59210]: 2024-03-04 16:07:13.834621141 +0000 UTC m=+0.137118936 volume create ee22d7beeb4d9010368196d51bffc47c4868fbc7b4a84f96d1e3b920a126d8e4 2024-03-04T16:07:14.128 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:13 smithi083 podman[59210]: 2024-03-04 16:07:13.951884415 +0000 UTC m=+0.254382207 container create 3c6dc3779a0eabacd3304932ed447e199081f1e0bcebe25785ac102ad244df79 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:07:14.488 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:14 smithi083 podman[59210]: 2024-03-04 16:07:14.230276005 +0000 UTC m=+0.532773810 container init 3c6dc3779a0eabacd3304932ed447e199081f1e0bcebe25785ac102ad244df79 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:07:14.488 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:14 smithi083 podman[59210]: 2024-03-04 16:07:14.233082662 +0000 UTC m=+0.535580455 container start 3c6dc3779a0eabacd3304932ed447e199081f1e0bcebe25785ac102ad244df79 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:07:14.488 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:14 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=info ts=2024-03-04T16:07:14.243Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-04T16:07:14.488 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:14 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=info ts=2024-03-04T16:07:14.244Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-04T16:07:14.488 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:14 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=info ts=2024-03-04T16:07:14.245Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.83 port=9094 2024-03-04T16:07:14.489 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:14 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=info ts=2024-03-04T16:07:14.246Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-04T16:07:14.489 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:14 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=info ts=2024-03-04T16:07:14.271Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-04T16:07:14.489 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:14 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=info ts=2024-03-04T16:07:14.271Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-04T16:07:14.489 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:14 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=info ts=2024-03-04T16:07:14.274Z caller=main.go:518 msg=Listening address=:9093 2024-03-04T16:07:14.489 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:14 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=info ts=2024-03-04T16:07:14.274Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-04T16:07:14.489 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:14 smithi083 bash[59210]: 3c6dc3779a0eabacd3304932ed447e199081f1e0bcebe25785ac102ad244df79 2024-03-04T16:07:14.489 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:14 smithi083 systemd[1]: Started Ceph alertmanager.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:07:15.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:15 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:15.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:15 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:15.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:15 smithi086 ceph-mon[25584]: Reconfiguring iscsi.foo.smithi083.fnypow (dependencies changed)... 2024-03-04T16:07:15.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:15 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:07:15.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:15 smithi086 ceph-mon[25584]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:15.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:15 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:07:15.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:15 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:15.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:15 smithi086 ceph-mon[25584]: Reconfiguring daemon iscsi.foo.smithi083.fnypow on smithi083 2024-03-04T16:07:15.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:15 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:15.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:15 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:15.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:15 smithi083 ceph-mon[24738]: Reconfiguring iscsi.foo.smithi083.fnypow (dependencies changed)... 2024-03-04T16:07:15.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:15 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:07:15.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:15 smithi083 ceph-mon[24738]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:15.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:15 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:07:15.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:15 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:15.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:15 smithi083 ceph-mon[24738]: Reconfiguring daemon iscsi.foo.smithi083.fnypow on smithi083 2024-03-04T16:07:15.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:15 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:15.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:15 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:15.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:15 smithi083 ceph-mon[29452]: Reconfiguring iscsi.foo.smithi083.fnypow (dependencies changed)... 2024-03-04T16:07:15.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:15 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:07:15.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:15 smithi083 ceph-mon[29452]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:15.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:15 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:07:15.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:15 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:15.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:15 smithi083 ceph-mon[29452]: Reconfiguring daemon iscsi.foo.smithi083.fnypow on smithi083 2024-03-04T16:07:16.137 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:15 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:07:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:07:16.488 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:16 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=info ts=2024-03-04T16:07:16.246Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000104085s 2024-03-04T16:07:17.632 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:17 smithi083 ceph-mon[24738]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:17.632 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:17 smithi083 ceph-mon[29452]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:17.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:17 smithi086 ceph-mon[25584]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:17.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:17 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:07:17] "GET /metrics HTTP/1.1" 200 38140 "" "Prometheus/2.33.4" 2024-03-04T16:07:18.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:07:18.142+0000 7fc74f87a700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi083.fnypow ... 2024-03-04T16:07:18.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Creating ceph-iscsi config... 2024-03-04T16:07:18.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/iscsi-gateway.cfg 2024-03-04T16:07:18.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/tcmu-runner-entrypoint.sh 2024-03-04T16:07:18.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Non-zero exit code 1 from systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow 2024-03-04T16:07:18.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: systemctl: stderr Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:07:18.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: systemctl: stderr See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:07:18.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Traceback (most recent call last): 2024-03-04T16:07:18.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-04T16:07:18.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: return _run_code(code, main_globals, None, 2024-03-04T16:07:18.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-04T16:07:18.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: exec(code, run_globals) 2024-03-04T16:07:18.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5527, in 2024-03-04T16:07:18.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5515, in main 2024-03-04T16:07:18.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3043, in command_deploy_from 2024-03-04T16:07:18.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3078, in _common_deploy 2024-03-04T16:07:18.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3098, in _deploy_daemon_container 2024-03-04T16:07:18.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 1070, in deploy_daemon 2024-03-04T16:07:18.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-04T16:07:18.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: RuntimeError: Failed command: systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow: Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:07:18.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:07:18.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Traceback (most recent call last): 2024-03-04T16:07:18.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-04T16:07:18.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-04T16:07:18.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2382, in _daemon_action 2024-03-04T16:07:18.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-04T16:07:18.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-04T16:07:18.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: return self.event_loop.get_result(coro, timeout) 2024-03-04T16:07:18.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-04T16:07:18.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: return future.result(timeout) 2024-03-04T16:07:18.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-04T16:07:18.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: return self.__get_result() 2024-03-04T16:07:18.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-04T16:07:18.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: raise self._exception 2024-03-04T16:07:18.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-04T16:07:18.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: config_blobs=daemon_spec.final_config, 2024-03-04T16:07:18.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-04T16:07:18.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-04T16:07:18.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi083.fnypow ... 2024-03-04T16:07:18.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Creating ceph-iscsi config... 2024-03-04T16:07:18.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/iscsi-gateway.cfg 2024-03-04T16:07:18.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/tcmu-runner-entrypoint.sh 2024-03-04T16:07:18.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Non-zero exit code 1 from systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow 2024-03-04T16:07:18.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: systemctl: stderr Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:07:18.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: systemctl: stderr See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:07:18.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Traceback (most recent call last): 2024-03-04T16:07:18.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-04T16:07:18.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: return _run_code(code, main_globals, None, 2024-03-04T16:07:18.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-04T16:07:18.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: exec(code, run_globals) 2024-03-04T16:07:18.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5527, in 2024-03-04T16:07:18.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5515, in main 2024-03-04T16:07:18.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3043, in command_deploy_from 2024-03-04T16:07:18.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3078, in _common_deploy 2024-03-04T16:07:18.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3098, in _deploy_daemon_container 2024-03-04T16:07:18.490 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 1070, in deploy_daemon 2024-03-04T16:07:18.490 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-04T16:07:18.490 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: RuntimeError: Failed command: systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow: Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:07:18.490 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:07:19.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:19 smithi086 ceph-mon[25584]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi083.fnypow ... 2024-03-04T16:07:19.734 INFO:journalctl@ceph.mon.b.smithi086.stdout: Creating ceph-iscsi config... 2024-03-04T16:07:19.735 INFO:journalctl@ceph.mon.b.smithi086.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/iscsi-gateway.cfg 2024-03-04T16:07:19.735 INFO:journalctl@ceph.mon.b.smithi086.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/tcmu-runner-entrypoint.sh 2024-03-04T16:07:19.735 INFO:journalctl@ceph.mon.b.smithi086.stdout: Non-zero exit code 1 from systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow 2024-03-04T16:07:19.735 INFO:journalctl@ceph.mon.b.smithi086.stdout: systemctl: stderr Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:07:19.735 INFO:journalctl@ceph.mon.b.smithi086.stdout: systemctl: stderr See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:07:19.735 INFO:journalctl@ceph.mon.b.smithi086.stdout: Traceback (most recent call last): 2024-03-04T16:07:19.735 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-04T16:07:19.735 INFO:journalctl@ceph.mon.b.smithi086.stdout: return _run_code(code, main_globals, None, 2024-03-04T16:07:19.735 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-04T16:07:19.735 INFO:journalctl@ceph.mon.b.smithi086.stdout: exec(code, run_globals) 2024-03-04T16:07:19.735 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5527, in 2024-03-04T16:07:19.735 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5515, in main 2024-03-04T16:07:19.735 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3043, in command_deploy_from 2024-03-04T16:07:19.736 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3078, in _common_deploy 2024-03-04T16:07:19.736 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3098, in _deploy_daemon_container 2024-03-04T16:07:19.736 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 1070, in deploy_daemon 2024-03-04T16:07:19.736 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-04T16:07:19.736 INFO:journalctl@ceph.mon.b.smithi086.stdout: RuntimeError: Failed command: systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow: Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:07:19.736 INFO:journalctl@ceph.mon.b.smithi086.stdout: See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:07:19.736 INFO:journalctl@ceph.mon.b.smithi086.stdout: Traceback (most recent call last): 2024-03-04T16:07:19.736 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-04T16:07:19.736 INFO:journalctl@ceph.mon.b.smithi086.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-04T16:07:19.736 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2382, in _daemon_action 2024-03-04T16:07:19.736 INFO:journalctl@ceph.mon.b.smithi086.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-04T16:07:19.736 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-04T16:07:19.736 INFO:journalctl@ceph.mon.b.smithi086.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-04T16:07:19.736 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-04T16:07:19.737 INFO:journalctl@ceph.mon.b.smithi086.stdout: return future.result(timeout) 2024-03-04T16:07:19.737 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-04T16:07:19.737 INFO:journalctl@ceph.mon.b.smithi086.stdout: return self.__get_result() 2024-03-04T16:07:19.737 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-04T16:07:19.737 INFO:journalctl@ceph.mon.b.smithi086.stdout: raise self._exception 2024-03-04T16:07:19.737 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-04T16:07:19.737 INFO:journalctl@ceph.mon.b.smithi086.stdout: config_blobs=daemon_spec.final_config, 2024-03-04T16:07:19.737 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-04T16:07:19.737 INFO:journalctl@ceph.mon.b.smithi086.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-04T16:07:19.737 INFO:journalctl@ceph.mon.b.smithi086.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi083.fnypow ... 2024-03-04T16:07:19.737 INFO:journalctl@ceph.mon.b.smithi086.stdout: Creating ceph-iscsi config... 2024-03-04T16:07:19.737 INFO:journalctl@ceph.mon.b.smithi086.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/iscsi-gateway.cfg 2024-03-04T16:07:19.737 INFO:journalctl@ceph.mon.b.smithi086.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/tcmu-runner-entrypoint.sh 2024-03-04T16:07:19.737 INFO:journalctl@ceph.mon.b.smithi086.stdout: Non-zero exit code 1 from systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow 2024-03-04T16:07:19.738 INFO:journalctl@ceph.mon.b.smithi086.stdout: systemctl: stderr Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:07:19.738 INFO:journalctl@ceph.mon.b.smithi086.stdout: systemctl: stderr See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:07:19.738 INFO:journalctl@ceph.mon.b.smithi086.stdout: Traceback (most recent call last): 2024-03-04T16:07:19.738 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-04T16:07:19.738 INFO:journalctl@ceph.mon.b.smithi086.stdout: return _run_code(code, main_globals, None, 2024-03-04T16:07:19.738 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-04T16:07:19.738 INFO:journalctl@ceph.mon.b.smithi086.stdout: exec(code, run_globals) 2024-03-04T16:07:19.738 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5527, in 2024-03-04T16:07:19.738 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5515, in main 2024-03-04T16:07:19.738 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3043, in command_deploy_from 2024-03-04T16:07:19.738 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3078, in _common_deploy 2024-03-04T16:07:19.738 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3098, in _deploy_daemon_container 2024-03-04T16:07:19.738 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 1070, in deploy_daemon 2024-03-04T16:07:19.738 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-04T16:07:19.739 INFO:journalctl@ceph.mon.b.smithi086.stdout: RuntimeError: Failed command: systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow: Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:07:19.739 INFO:journalctl@ceph.mon.b.smithi086.stdout: See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:07:19.739 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:19 smithi086 ceph-mon[25584]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-04T16:07:19.739 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:19 smithi086 ceph-mon[25584]: Reconfiguring daemon prometheus.a on smithi086 2024-03-04T16:07:19.739 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:19 smithi086 ceph-mon[25584]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:07:19.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:19 smithi083 ceph-mon[29452]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi083.fnypow ... 2024-03-04T16:07:19.739 INFO:journalctl@ceph.mon.c.smithi083.stdout: Creating ceph-iscsi config... 2024-03-04T16:07:19.739 INFO:journalctl@ceph.mon.c.smithi083.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/iscsi-gateway.cfg 2024-03-04T16:07:19.740 INFO:journalctl@ceph.mon.c.smithi083.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/tcmu-runner-entrypoint.sh 2024-03-04T16:07:19.740 INFO:journalctl@ceph.mon.c.smithi083.stdout: Non-zero exit code 1 from systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow 2024-03-04T16:07:19.740 INFO:journalctl@ceph.mon.c.smithi083.stdout: systemctl: stderr Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:07:19.740 INFO:journalctl@ceph.mon.c.smithi083.stdout: systemctl: stderr See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:07:19.740 INFO:journalctl@ceph.mon.c.smithi083.stdout: Traceback (most recent call last): 2024-03-04T16:07:19.740 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-04T16:07:19.740 INFO:journalctl@ceph.mon.c.smithi083.stdout: return _run_code(code, main_globals, None, 2024-03-04T16:07:19.740 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-04T16:07:19.740 INFO:journalctl@ceph.mon.c.smithi083.stdout: exec(code, run_globals) 2024-03-04T16:07:19.740 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5527, in 2024-03-04T16:07:19.740 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5515, in main 2024-03-04T16:07:19.740 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3043, in command_deploy_from 2024-03-04T16:07:19.740 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3078, in _common_deploy 2024-03-04T16:07:19.741 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3098, in _deploy_daemon_container 2024-03-04T16:07:19.741 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 1070, in deploy_daemon 2024-03-04T16:07:19.741 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-04T16:07:19.741 INFO:journalctl@ceph.mon.c.smithi083.stdout: RuntimeError: Failed command: systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow: Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:07:19.741 INFO:journalctl@ceph.mon.c.smithi083.stdout: See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:07:19.741 INFO:journalctl@ceph.mon.c.smithi083.stdout: Traceback (most recent call last): 2024-03-04T16:07:19.741 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-04T16:07:19.741 INFO:journalctl@ceph.mon.c.smithi083.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-04T16:07:19.741 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2382, in _daemon_action 2024-03-04T16:07:19.741 INFO:journalctl@ceph.mon.c.smithi083.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-04T16:07:19.741 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-04T16:07:19.741 INFO:journalctl@ceph.mon.c.smithi083.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-04T16:07:19.741 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-04T16:07:19.741 INFO:journalctl@ceph.mon.c.smithi083.stdout: return future.result(timeout) 2024-03-04T16:07:19.741 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-04T16:07:19.742 INFO:journalctl@ceph.mon.c.smithi083.stdout: return self.__get_result() 2024-03-04T16:07:19.742 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-04T16:07:19.742 INFO:journalctl@ceph.mon.c.smithi083.stdout: raise self._exception 2024-03-04T16:07:19.742 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-04T16:07:19.742 INFO:journalctl@ceph.mon.c.smithi083.stdout: config_blobs=daemon_spec.final_config, 2024-03-04T16:07:19.742 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-04T16:07:19.742 INFO:journalctl@ceph.mon.c.smithi083.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-04T16:07:19.742 INFO:journalctl@ceph.mon.c.smithi083.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi083.fnypow ... 2024-03-04T16:07:19.742 INFO:journalctl@ceph.mon.c.smithi083.stdout: Creating ceph-iscsi config... 2024-03-04T16:07:19.742 INFO:journalctl@ceph.mon.c.smithi083.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/iscsi-gateway.cfg 2024-03-04T16:07:19.742 INFO:journalctl@ceph.mon.c.smithi083.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/tcmu-runner-entrypoint.sh 2024-03-04T16:07:19.742 INFO:journalctl@ceph.mon.c.smithi083.stdout: Non-zero exit code 1 from systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow 2024-03-04T16:07:19.742 INFO:journalctl@ceph.mon.c.smithi083.stdout: systemctl: stderr Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:07:19.742 INFO:journalctl@ceph.mon.c.smithi083.stdout: systemctl: stderr See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:07:19.742 INFO:journalctl@ceph.mon.c.smithi083.stdout: Traceback (most recent call last): 2024-03-04T16:07:19.743 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-04T16:07:19.743 INFO:journalctl@ceph.mon.c.smithi083.stdout: return _run_code(code, main_globals, None, 2024-03-04T16:07:19.743 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-04T16:07:19.743 INFO:journalctl@ceph.mon.c.smithi083.stdout: exec(code, run_globals) 2024-03-04T16:07:19.743 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5527, in 2024-03-04T16:07:19.743 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5515, in main 2024-03-04T16:07:19.743 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3043, in command_deploy_from 2024-03-04T16:07:19.743 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3078, in _common_deploy 2024-03-04T16:07:19.743 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:19 smithi083 ceph-mon[24738]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi083.fnypow ... 2024-03-04T16:07:19.744 INFO:journalctl@ceph.mon.a.smithi083.stdout: Creating ceph-iscsi config... 2024-03-04T16:07:19.744 INFO:journalctl@ceph.mon.a.smithi083.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/iscsi-gateway.cfg 2024-03-04T16:07:19.744 INFO:journalctl@ceph.mon.a.smithi083.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/tcmu-runner-entrypoint.sh 2024-03-04T16:07:19.744 INFO:journalctl@ceph.mon.a.smithi083.stdout: Non-zero exit code 1 from systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow 2024-03-04T16:07:19.744 INFO:journalctl@ceph.mon.a.smithi083.stdout: systemctl: stderr Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:07:19.744 INFO:journalctl@ceph.mon.a.smithi083.stdout: systemctl: stderr See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:07:19.744 INFO:journalctl@ceph.mon.a.smithi083.stdout: Traceback (most recent call last): 2024-03-04T16:07:19.744 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-04T16:07:19.744 INFO:journalctl@ceph.mon.a.smithi083.stdout: return _run_code(code, main_globals, None, 2024-03-04T16:07:19.744 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-04T16:07:19.744 INFO:journalctl@ceph.mon.a.smithi083.stdout: exec(code, run_globals) 2024-03-04T16:07:19.744 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5527, in 2024-03-04T16:07:19.744 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5515, in main 2024-03-04T16:07:19.744 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3043, in command_deploy_from 2024-03-04T16:07:19.745 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3078, in _common_deploy 2024-03-04T16:07:19.745 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3098, in _deploy_daemon_container 2024-03-04T16:07:19.745 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 1070, in deploy_daemon 2024-03-04T16:07:19.745 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-04T16:07:19.745 INFO:journalctl@ceph.mon.a.smithi083.stdout: RuntimeError: Failed command: systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow: Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:07:19.745 INFO:journalctl@ceph.mon.a.smithi083.stdout: See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:07:19.745 INFO:journalctl@ceph.mon.a.smithi083.stdout: Traceback (most recent call last): 2024-03-04T16:07:19.745 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-04T16:07:19.745 INFO:journalctl@ceph.mon.a.smithi083.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-04T16:07:19.745 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2382, in _daemon_action 2024-03-04T16:07:19.745 INFO:journalctl@ceph.mon.a.smithi083.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-04T16:07:19.745 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-04T16:07:19.745 INFO:journalctl@ceph.mon.a.smithi083.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-04T16:07:19.745 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-04T16:07:19.746 INFO:journalctl@ceph.mon.a.smithi083.stdout: return future.result(timeout) 2024-03-04T16:07:19.746 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-04T16:07:19.746 INFO:journalctl@ceph.mon.a.smithi083.stdout: return self.__get_result() 2024-03-04T16:07:19.746 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-04T16:07:19.746 INFO:journalctl@ceph.mon.a.smithi083.stdout: raise self._exception 2024-03-04T16:07:19.746 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-04T16:07:19.746 INFO:journalctl@ceph.mon.a.smithi083.stdout: config_blobs=daemon_spec.final_config, 2024-03-04T16:07:19.746 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-04T16:07:19.746 INFO:journalctl@ceph.mon.a.smithi083.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-04T16:07:19.746 INFO:journalctl@ceph.mon.a.smithi083.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi083.fnypow ... 2024-03-04T16:07:19.746 INFO:journalctl@ceph.mon.a.smithi083.stdout: Creating ceph-iscsi config... 2024-03-04T16:07:19.746 INFO:journalctl@ceph.mon.a.smithi083.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/iscsi-gateway.cfg 2024-03-04T16:07:19.746 INFO:journalctl@ceph.mon.a.smithi083.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/tcmu-runner-entrypoint.sh 2024-03-04T16:07:19.746 INFO:journalctl@ceph.mon.a.smithi083.stdout: Non-zero exit code 1 from systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow 2024-03-04T16:07:19.746 INFO:journalctl@ceph.mon.a.smithi083.stdout: systemctl: stderr Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:07:19.747 INFO:journalctl@ceph.mon.a.smithi083.stdout: systemctl: stderr See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:07:19.747 INFO:journalctl@ceph.mon.a.smithi083.stdout: Traceback (most recent call last): 2024-03-04T16:07:19.747 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-04T16:07:19.747 INFO:journalctl@ceph.mon.a.smithi083.stdout: return _run_code(code, main_globals, None, 2024-03-04T16:07:19.747 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-04T16:07:19.747 INFO:journalctl@ceph.mon.a.smithi083.stdout: exec(code, run_globals) 2024-03-04T16:07:19.747 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5527, in 2024-03-04T16:07:19.747 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5515, in main 2024-03-04T16:07:19.747 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3043, in command_deploy_from 2024-03-04T16:07:19.747 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3078, in _common_deploy 2024-03-04T16:07:19.747 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3098, in _deploy_daemon_container 2024-03-04T16:07:19.747 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 1070, in deploy_daemon 2024-03-04T16:07:19.747 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-04T16:07:19.747 INFO:journalctl@ceph.mon.a.smithi083.stdout: RuntimeError: Failed command: systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow: Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:07:19.747 INFO:journalctl@ceph.mon.a.smithi083.stdout: See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:07:19.747 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:19 smithi083 ceph-mon[24738]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-04T16:07:19.748 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:19 smithi083 ceph-mon[24738]: Reconfiguring daemon prometheus.a on smithi086 2024-03-04T16:07:19.748 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:19 smithi083 ceph-mon[24738]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:07:19.748 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3098, in _deploy_daemon_container 2024-03-04T16:07:19.748 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 1070, in deploy_daemon 2024-03-04T16:07:19.748 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-04T16:07:19.748 INFO:journalctl@ceph.mon.c.smithi083.stdout: RuntimeError: Failed command: systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow: Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:07:19.748 INFO:journalctl@ceph.mon.c.smithi083.stdout: See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:07:19.748 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:19 smithi083 ceph-mon[29452]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-04T16:07:19.748 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:19 smithi083 ceph-mon[29452]: Reconfiguring daemon prometheus.a on smithi086 2024-03-04T16:07:19.748 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:19 smithi083 ceph-mon[29452]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:07:21.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:21 smithi086 ceph-mon[25584]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-04T16:07:21.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:21 smithi083 ceph-mon[24738]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-04T16:07:21.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:21 smithi083 ceph-mon[29452]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-04T16:07:22.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:22 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:07:22.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.86:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.83:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:07:23.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:23 smithi086 ceph-mon[25584]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-04T16:07:23.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:23 smithi083 ceph-mon[24738]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-04T16:07:23.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:23 smithi083 ceph-mon[29452]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-04T16:07:24.738 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:07:24 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=info ts=2024-03-04T16:07:24.248Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.00210273s 2024-03-04T16:07:25.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:25 smithi086 ceph-mon[25584]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:07:25.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:25 smithi083 ceph-mon[29452]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:07:25.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:25 smithi083 ceph-mon[24738]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:07:26.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:25 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[24970]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:07:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:07:26.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:26 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:26.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:26 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:26.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:26 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:27.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:27 smithi086 ceph-mon[25584]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:07:27.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:07:27] "GET /metrics HTTP/1.1" 200 38141 "" "Prometheus/2.33.4" 2024-03-04T16:07:27.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:27 smithi083 ceph-mon[24738]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:07:27.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:27 smithi083 ceph-mon[29452]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:07:29.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:29 smithi083 ceph-mon[24738]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:07:29.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:29 smithi083 ceph-mon[29452]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:07:29.799 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:29 smithi086 ceph-mon[25584]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:07:30.112 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:29 smithi086 systemd[1]: Stopping Ceph prometheus.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:07:30.113 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:29 smithi086 bash[50271]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus.a" found: no such container 2024-03-04T16:07:30.410 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:07:30.112Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-04T16:07:30.410 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:07:30.112Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-04T16:07:30.411 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:07:30.112Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-04T16:07:30.411 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:07:30.112Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-04T16:07:30.411 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:07:30.112Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-04T16:07:30.411 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:07:30.112Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-04T16:07:30.411 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:07:30.112Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-04T16:07:30.411 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:07:30.112Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-04T16:07:30.411 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:07:30.112Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-04T16:07:30.411 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:07:30.113Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-04T16:07:30.411 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:07:30.113Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-04T16:07:30.411 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[42406]: ts=2024-03-04T16:07:30.113Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-04T16:07:30.411 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 podman[50284]: 2024-03-04 16:07:30.122620307 +0000 UTC m=+0.119416396 container died 82e50b4a8b864d533e268dba33b9e5487364e8c28580a310eeca80033a6f5019 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:07:30.473 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:07:30.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:30 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/434223007' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T16:07:30.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:30 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1619330225' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1809332442"}]: dispatch 2024-03-04T16:07:30.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 podman[50284]: 2024-03-04 16:07:30.503701813 +0000 UTC m=+0.500497884 container cleanup 82e50b4a8b864d533e268dba33b9e5487364e8c28580a310eeca80033a6f5019 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:07:30.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 bash[50284]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a 2024-03-04T16:07:30.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 bash[50312]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus.a" found: no such container 2024-03-04T16:07:30.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:30 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/434223007' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T16:07:30.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:30 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1619330225' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1809332442"}]: dispatch 2024-03-04T16:07:30.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:30 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/434223007' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T16:07:30.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:30 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1619330225' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1809332442"}]: dispatch 2024-03-04T16:07:31.014 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 podman[50298]: 2024-03-04 16:07:30.754911367 +0000 UTC m=+0.634477525 container remove 82e50b4a8b864d533e268dba33b9e5487364e8c28580a310eeca80033a6f5019 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:07:31.014 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-04T16:07:31.014 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 systemd[1]: Stopped Ceph prometheus.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:07:31.014 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@prometheus.a.service: Consumed 1.511s CPU time. 2024-03-04T16:07:31.014 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 systemd[1]: Starting Ceph prometheus.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:07:31.014 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:30 smithi086 podman[50381]: 2024-03-04 16:07:30.882264939 +0000 UTC m=+0.019115424 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-04T16:07:31.014 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 podman[50381]: 2024-03-04 16:07:31.014362602 +0000 UTC m=+0.151213078 container create f2ee5ff207554606a69c14ddd5e88f9b4abedfb85c1eaa9c0121c3ed2c3ce254 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:07:31.109 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:07:31.109 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 starting - - - - 2024-03-04T16:07:31.109 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (5m) 36s ago 5m 53.8M - 8.3.5 dad864ee21e9 9c0f7c6afe4b 2024-03-04T16:07:31.109 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 running (4m) 41s ago 4m 53.4M - 3.5 e1d6a67b021e 553466e95b30 2024-03-04T16:07:31.109 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443,9283 running (59s) 36s ago 9m 490M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e a73348a7968e 2024-03-04T16:07:31.109 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:9283 running (11m) 41s ago 11m 408M - 17.2.0 e1d6a67b021e 5af09a7ff74e 2024-03-04T16:07:31.109 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (11m) 41s ago 11m 50.2M 2048M 17.2.0 e1d6a67b021e d2ba878ed364 2024-03-04T16:07:31.109 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (10m) 36s ago 10m 39.6M 2048M 17.2.0 e1d6a67b021e d1b972ebdd2a 2024-03-04T16:07:31.109 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (10m) 41s ago 10m 39.7M 2048M 17.2.0 e1d6a67b021e e74b0ab73a80 2024-03-04T16:07:31.109 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 starting - - - - 2024-03-04T16:07:31.110 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (5m) 36s ago 5m 16.6M - 1.3.1 1dbe0e931976 9091c36c27a4 2024-03-04T16:07:31.110 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (9m) 41s ago 9m 51.0M 2736M 17.2.0 e1d6a67b021e 56679a9f787b 2024-03-04T16:07:31.110 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (9m) 41s ago 9m 55.3M 2736M 17.2.0 e1d6a67b021e c1070c1a315a 2024-03-04T16:07:31.110 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (8m) 41s ago 8m 47.2M 2736M 17.2.0 e1d6a67b021e 2b913c19f581 2024-03-04T16:07:31.110 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (8m) 41s ago 8m 49.3M 2736M 17.2.0 e1d6a67b021e 9f9a4e2ec5da 2024-03-04T16:07:31.110 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (8m) 36s ago 8m 51.5M 2992M 17.2.0 e1d6a67b021e b3a97f83f1c7 2024-03-04T16:07:31.110 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (7m) 36s ago 7m 49.6M 2992M 17.2.0 e1d6a67b021e b82ca42b7353 2024-03-04T16:07:31.110 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (7m) 36s ago 7m 48.0M 2992M 17.2.0 e1d6a67b021e 18cedf14321d 2024-03-04T16:07:31.110 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (6m) 36s ago 6m 52.2M 2992M 17.2.0 e1d6a67b021e 390e1798c738 2024-03-04T16:07:31.110 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 running (4m) 36s ago 6m 66.6M - 2.33.4 514e6a882f6e 82e50b4a8b86 2024-03-04T16:07:31.110 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (4m) 41s ago 4m 72.0M - 17.2.0 e1d6a67b021e 65f14042f354 2024-03-04T16:07:31.110 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (4m) 36s ago 4m 73.0M - 17.2.0 e1d6a67b021e 78175cc6dc5d 2024-03-04T16:07:31.110 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (4m) 41s ago 4m 70.2M - 17.2.0 e1d6a67b021e 01dcf858bd62 2024-03-04T16:07:31.110 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (4m) 36s ago 4m 75.0M - 17.2.0 e1d6a67b021e 8f70a5ce9f0d 2024-03-04T16:07:31.284 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 podman[50381]: 2024-03-04 16:07:31.25859579 +0000 UTC m=+0.395446267 container init f2ee5ff207554606a69c14ddd5e88f9b4abedfb85c1eaa9c0121c3ed2c3ce254 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:07:31.284 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 podman[50381]: 2024-03-04 16:07:31.261399998 +0000 UTC m=+0.398250481 container start f2ee5ff207554606a69c14ddd5e88f9b4abedfb85c1eaa9c0121c3ed2c3ce254 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:07:31.519 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:07:31.519 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:07:31.519 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-04T16:07:31.519 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:07:31.519 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:07:31.519 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-04T16:07:31.519 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 1 2024-03-04T16:07:31.519 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:07:31.519 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:07:31.519 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-04T16:07:31.519 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:07:31.519 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": {}, 2024-03-04T16:07:31.519 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:07:31.520 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-04T16:07:31.520 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:07:31.520 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:07:31.520 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-03-04T16:07:31.520 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 1 2024-03-04T16:07:31.520 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:07:31.520 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:07:31.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-mon[25584]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:07:31.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-mon[25584]: from='client.25159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:31.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1619330225' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1809332442"}]': finished 2024-03-04T16:07:31.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-mon[25584]: osdmap e98: 8 total, 8 up, 8 in 2024-03-04T16:07:31.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:31.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:31.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.284Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-04T16:07:31.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.284Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-04T16:07:31.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.284Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-04T16:07:31.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.284Z caller=main.go:518 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi086 (none))" 2024-03-04T16:07:31.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.284Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-04T16:07:31.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.284Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-04T16:07:31.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.286Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-04T16:07:31.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.286Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-04T16:07:31.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.287Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-04T16:07:31.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.289Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-04T16:07:31.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.289Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.568µs 2024-03-04T16:07:31.736 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.289Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-04T16:07:31.736 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 bash[50381]: f2ee5ff207554606a69c14ddd5e88f9b4abedfb85c1eaa9c0121c3ed2c3ce254 2024-03-04T16:07:31.736 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 systemd[1]: Started Ceph prometheus.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:07:31.736 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.495Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-03-04T16:07:31.839 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:07:31.840 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:07:31.840 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:07:31.840 INFO:teuthology.orchestra.run.smithi083.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:07:31.840 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [], 2024-03-04T16:07:31.840 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "1/23 daemons upgraded", 2024-03-04T16:07:31.840 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "", 2024-03-04T16:07:31.840 INFO:teuthology.orchestra.run.smithi083.stdout: "is_paused": false 2024-03-04T16:07:31.840 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:07:31.841 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:31 smithi083 ceph-mon[24738]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:07:31.841 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:31 smithi083 ceph-mon[24738]: from='client.25159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:31.841 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:31 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1619330225' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1809332442"}]': finished 2024-03-04T16:07:31.841 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:31 smithi083 ceph-mon[24738]: osdmap e98: 8 total, 8 up, 8 in 2024-03-04T16:07:31.841 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:31 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:31.841 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:31 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:31.841 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:31 smithi083 ceph-mon[29452]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:07:31.841 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:31 smithi083 ceph-mon[29452]: from='client.25159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:31.841 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:31 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1619330225' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1809332442"}]': finished 2024-03-04T16:07:31.841 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:31 smithi083 ceph-mon[29452]: osdmap e98: 8 total, 8 up, 8 in 2024-03-04T16:07:31.841 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:31 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:31.841 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:31 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:32.054 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:31 smithi086 systemd[1]: Stopping Ceph node-exporter.b for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:07:32.055 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:31 smithi086 bash[50503]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter.b" found: no such container 2024-03-04T16:07:32.055 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.743Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-03-04T16:07:32.055 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.743Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-03-04T16:07:32.055 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.743Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=27.559µs wal_replay_duration=453.914448ms total_replay_duration=453.95552ms 2024-03-04T16:07:32.055 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.748Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-04T16:07:32.055 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.748Z caller=main.go:947 level=info msg="TSDB started" 2024-03-04T16:07:32.055 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.748Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-04T16:07:32.055 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.776Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=28.079458ms db_storage=821ns remote_storage=1.981µs web_handler=354ns query_engine=695ns scrape=614.09µs scrape_sd=161.886µs notify=24.022µs notify_sd=17.384µs rules=26.822848ms 2024-03-04T16:07:32.056 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:31 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:31.776Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-04T16:07:32.219 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_OK 2024-03-04T16:07:32.345 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:32 smithi086 podman[50514]: 2024-03-04 16:07:32.055045397 +0000 UTC m=+0.115100632 container died 9091c36c27a4a6270f0a45246698d1ea4342e0b6f4c325501f110c7daa425ea3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T16:07:32.663 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:32 smithi086 podman[50514]: 2024-03-04 16:07:32.444578277 +0000 UTC m=+0.504633510 container cleanup 9091c36c27a4a6270f0a45246698d1ea4342e0b6f4c325501f110c7daa425ea3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T16:07:32.663 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:32 smithi086 bash[50514]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b 2024-03-04T16:07:32.663 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:32 smithi086 bash[50542]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter.b" found: no such container 2024-03-04T16:07:32.663 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:32 smithi086 ceph-mon[25584]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:32.663 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:32 smithi086 ceph-mon[25584]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:32.664 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:32 smithi086 ceph-mon[25584]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-04T16:07:32.664 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:32 smithi086 ceph-mon[25584]: Reconfiguring daemon node-exporter.b on smithi086 2024-03-04T16:07:32.664 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:32 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3628697881' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:32.664 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:32 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1834129884' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/853568958"}]: dispatch 2024-03-04T16:07:32.664 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:32 smithi086 ceph-mon[25584]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/853568958"}]: dispatch 2024-03-04T16:07:32.664 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:32 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1985884816' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:07:32.946 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:32 smithi086 podman[50529]: 2024-03-04 16:07:32.704722834 +0000 UTC m=+0.653983996 container remove 9091c36c27a4a6270f0a45246698d1ea4342e0b6f4c325501f110c7daa425ea3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T16:07:32.946 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:32 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-04T16:07:32.946 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:32 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-04T16:07:32.946 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:32 smithi086 systemd[1]: Stopped Ceph node-exporter.b for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:07:32.946 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:32 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@node-exporter.b.service: Consumed 2.852s CPU time. 2024-03-04T16:07:32.946 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:32 smithi086 systemd[1]: Starting Ceph node-exporter.b for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:07:32.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:32 smithi083 ceph-mon[24738]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:32.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:32 smithi083 ceph-mon[24738]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:32.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:32 smithi083 ceph-mon[24738]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-04T16:07:32.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:32 smithi083 ceph-mon[24738]: Reconfiguring daemon node-exporter.b on smithi086 2024-03-04T16:07:32.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:32 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3628697881' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:32.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:32 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1834129884' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/853568958"}]: dispatch 2024-03-04T16:07:32.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:32 smithi083 ceph-mon[24738]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/853568958"}]: dispatch 2024-03-04T16:07:32.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:32 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1985884816' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:07:32.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:32 smithi083 ceph-mon[29452]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:32.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:32 smithi083 ceph-mon[29452]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:32.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:32 smithi083 ceph-mon[29452]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-04T16:07:32.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:32 smithi083 ceph-mon[29452]: Reconfiguring daemon node-exporter.b on smithi086 2024-03-04T16:07:32.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:32 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3628697881' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:07:32.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:32 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1834129884' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/853568958"}]: dispatch 2024-03-04T16:07:32.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:32 smithi083 ceph-mon[29452]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/853568958"}]: dispatch 2024-03-04T16:07:32.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:32 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1985884816' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:07:33.234 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:32 smithi086 podman[50618]: 2024-03-04 16:07:32.848841425 +0000 UTC m=+0.017850141 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-04T16:07:33.234 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:32 smithi086 podman[50618]: 2024-03-04 16:07:32.968326967 +0000 UTC m=+0.137335671 container create b72dd80b2588b252d353cccc20e29588869ea0b17d66ae09f32cda7d0c36ba15 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T16:07:33.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:33 smithi086 ceph-mon[25584]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:33.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:33 smithi086 ceph-mon[25584]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-04T16:07:33.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:33 smithi086 ceph-mon[25584]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/853568958"}]': finished 2024-03-04T16:07:33.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:33 smithi086 ceph-mon[25584]: osdmap e99: 8 total, 8 up, 8 in 2024-03-04T16:07:33.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:33 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/799194998' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1854650023"}]: dispatch 2024-03-04T16:07:33.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:33 smithi083 ceph-mon[24738]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:33.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:33 smithi083 ceph-mon[24738]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-04T16:07:33.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:33 smithi083 ceph-mon[24738]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/853568958"}]': finished 2024-03-04T16:07:33.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:33 smithi083 ceph-mon[24738]: osdmap e99: 8 total, 8 up, 8 in 2024-03-04T16:07:33.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:33 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/799194998' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1854650023"}]: dispatch 2024-03-04T16:07:33.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:33 smithi083 ceph-mon[29452]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:07:33.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:33 smithi083 ceph-mon[29452]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-04T16:07:33.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:33 smithi083 ceph-mon[29452]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/853568958"}]': finished 2024-03-04T16:07:33.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:33 smithi083 ceph-mon[29452]: osdmap e99: 8 total, 8 up, 8 in 2024-03-04T16:07:33.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:33 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/799194998' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1854650023"}]: dispatch 2024-03-04T16:07:34.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/799194998' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1854650023"}]': finished 2024-03-04T16:07:34.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-mon[25584]: osdmap e100: 8 total, 8 up, 8 in 2024-03-04T16:07:34.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3268986836' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2796988943"}]: dispatch 2024-03-04T16:07:34.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-mon[25584]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2796988943"}]: dispatch 2024-03-04T16:07:34.985 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 podman[50618]: 2024-03-04 16:07:34.667351406 +0000 UTC m=+1.836360110 container init b72dd80b2588b252d353cccc20e29588869ea0b17d66ae09f32cda7d0c36ba15 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T16:07:34.985 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 podman[50618]: 2024-03-04 16:07:34.670887471 +0000 UTC m=+1.839896175 container start b72dd80b2588b252d353cccc20e29588869ea0b17d66ae09f32cda7d0c36ba15 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T16:07:34.985 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.675Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-04T16:07:34.985 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.675Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-04T16:07:34.985 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-04T16:07:34.985 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-04T16:07:34.985 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-04T16:07:34.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=arp 2024-03-04T16:07:34.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-04T16:07:34.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-04T16:07:34.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-04T16:07:34.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-04T16:07:34.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-04T16:07:34.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-04T16:07:34.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-04T16:07:34.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-04T16:07:34.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=edac 2024-03-04T16:07:34.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-04T16:07:34.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-04T16:07:34.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-04T16:07:34.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-04T16:07:34.986 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-04T16:07:34.987 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-04T16:07:34.987 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-04T16:07:34.987 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-04T16:07:34.987 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-04T16:07:34.987 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-04T16:07:34.987 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-04T16:07:34.987 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-04T16:07:34.987 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-04T16:07:34.987 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-04T16:07:34.987 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-04T16:07:34.987 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-04T16:07:34.987 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=os 2024-03-04T16:07:34.987 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-04T16:07:34.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-04T16:07:34.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-04T16:07:34.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-04T16:07:34.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-04T16:07:34.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-04T16:07:34.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=stat 2024-03-04T16:07:34.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-04T16:07:34.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-04T16:07:34.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-04T16:07:34.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=time 2024-03-04T16:07:34.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-04T16:07:34.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=uname 2024-03-04T16:07:34.988 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-04T16:07:34.989 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-04T16:07:34.989 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-04T16:07:34.989 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.676Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-04T16:07:34.989 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b[50635]: ts=2024-03-04T16:07:34.677Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-04T16:07:34.989 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 bash[50618]: b72dd80b2588b252d353cccc20e29588869ea0b17d66ae09f32cda7d0c36ba15 2024-03-04T16:07:34.989 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:07:34 smithi086 systemd[1]: Started Ceph node-exporter.b for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:07:34.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:34 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/799194998' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1854650023"}]': finished 2024-03-04T16:07:34.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:34 smithi083 ceph-mon[24738]: osdmap e100: 8 total, 8 up, 8 in 2024-03-04T16:07:34.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:34 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3268986836' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2796988943"}]: dispatch 2024-03-04T16:07:34.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:34 smithi083 ceph-mon[24738]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2796988943"}]: dispatch 2024-03-04T16:07:34.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:34 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/799194998' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1854650023"}]': finished 2024-03-04T16:07:34.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:34 smithi083 ceph-mon[29452]: osdmap e100: 8 total, 8 up, 8 in 2024-03-04T16:07:34.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:34 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3268986836' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2796988943"}]: dispatch 2024-03-04T16:07:34.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:34 smithi083 ceph-mon[29452]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2796988943"}]: dispatch 2024-03-04T16:07:35.819 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:35 smithi083 ceph-mon[24738]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-04T16:07:35.820 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:35 smithi083 ceph-mon[24738]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2796988943"}]': finished 2024-03-04T16:07:35.820 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:35 smithi083 ceph-mon[24738]: osdmap e101: 8 total, 8 up, 8 in 2024-03-04T16:07:35.820 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:35 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:35.820 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:35 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:35.820 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:35 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3940874534' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/1219817995"}]: dispatch 2024-03-04T16:07:35.820 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:35 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:35.820 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:35 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:35.820 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:35 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T16:07:35.820 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:35 smithi083 ceph-mon[29452]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-04T16:07:35.820 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:35 smithi083 ceph-mon[29452]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2796988943"}]': finished 2024-03-04T16:07:35.820 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:35 smithi083 ceph-mon[29452]: osdmap e101: 8 total, 8 up, 8 in 2024-03-04T16:07:35.821 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:35 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:35.821 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:35 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:35.821 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:35 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3940874534' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/1219817995"}]: dispatch 2024-03-04T16:07:35.821 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:35 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:35.821 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:35 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:35.821 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:35 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T16:07:35.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:35 smithi086 ceph-mon[25584]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-04T16:07:35.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:35 smithi086 ceph-mon[25584]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2796988943"}]': finished 2024-03-04T16:07:35.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:35 smithi086 ceph-mon[25584]: osdmap e101: 8 total, 8 up, 8 in 2024-03-04T16:07:35.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:35 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:35.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:35 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:35.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:35 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3940874534' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/1219817995"}]: dispatch 2024-03-04T16:07:35.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:35 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:35.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:35 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:35.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:35 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T16:07:36.817 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:36 smithi083 ceph-mon[24738]: Reconfiguring grafana.a (dependencies changed)... 2024-03-04T16:07:36.817 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:36 smithi083 ceph-mon[24738]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-04T16:07:36.817 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:36 smithi083 ceph-mon[24738]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T16:07:36.817 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:36 smithi083 ceph-mon[24738]: Reconfiguring daemon grafana.a on smithi086 2024-03-04T16:07:36.817 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:36 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/3940874534' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/1219817995"}]': finished 2024-03-04T16:07:36.818 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:36 smithi083 ceph-mon[24738]: osdmap e102: 8 total, 8 up, 8 in 2024-03-04T16:07:36.818 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:36 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2777654143' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/1219817995"}]: dispatch 2024-03-04T16:07:36.818 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:36 smithi083 ceph-mon[29452]: Reconfiguring grafana.a (dependencies changed)... 2024-03-04T16:07:36.818 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:36 smithi083 ceph-mon[29452]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-04T16:07:36.818 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:36 smithi083 ceph-mon[29452]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T16:07:36.818 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:36 smithi083 ceph-mon[29452]: Reconfiguring daemon grafana.a on smithi086 2024-03-04T16:07:36.818 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:36 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3940874534' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/1219817995"}]': finished 2024-03-04T16:07:36.818 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:36 smithi083 ceph-mon[29452]: osdmap e102: 8 total, 8 up, 8 in 2024-03-04T16:07:36.818 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:36 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/2777654143' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/1219817995"}]: dispatch 2024-03-04T16:07:36.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:36 smithi086 ceph-mon[25584]: Reconfiguring grafana.a (dependencies changed)... 2024-03-04T16:07:36.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:36 smithi086 ceph-mon[25584]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-04T16:07:36.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:36 smithi086 ceph-mon[25584]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T16:07:36.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:36 smithi086 ceph-mon[25584]: Reconfiguring daemon grafana.a on smithi086 2024-03-04T16:07:36.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:36 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3940874534' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/1219817995"}]': finished 2024-03-04T16:07:36.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:36 smithi086 ceph-mon[25584]: osdmap e102: 8 total, 8 up, 8 in 2024-03-04T16:07:36.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:36 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/2777654143' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/1219817995"}]: dispatch 2024-03-04T16:07:37.825 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:37 smithi083 ceph-mon[29452]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:37.825 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:37 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/2777654143' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/1219817995"}]': finished 2024-03-04T16:07:37.825 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:37 smithi083 ceph-mon[29452]: osdmap e103: 8 total, 8 up, 8 in 2024-03-04T16:07:37.826 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:37 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/2870534703' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2940780554"}]: dispatch 2024-03-04T16:07:37.826 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:37 smithi083 ceph-mon[24738]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:37.826 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:37 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2777654143' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/1219817995"}]': finished 2024-03-04T16:07:37.826 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:37 smithi083 ceph-mon[24738]: osdmap e103: 8 total, 8 up, 8 in 2024-03-04T16:07:37.826 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:37 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2870534703' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2940780554"}]: dispatch 2024-03-04T16:07:37.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:37 smithi086 ceph-mon[25584]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:37.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:37 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/2777654143' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/1219817995"}]': finished 2024-03-04T16:07:37.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:37 smithi086 ceph-mon[25584]: osdmap e103: 8 total, 8 up, 8 in 2024-03-04T16:07:37.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:37 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/2870534703' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2940780554"}]: dispatch 2024-03-04T16:07:39.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:38 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/2870534703' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2940780554"}]': finished 2024-03-04T16:07:39.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:38 smithi086 ceph-mon[25584]: osdmap e104: 8 total, 8 up, 8 in 2024-03-04T16:07:39.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:38 smithi086 ceph-mon[25584]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:39.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:38 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/2870534703' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2940780554"}]': finished 2024-03-04T16:07:39.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:38 smithi083 ceph-mon[29452]: osdmap e104: 8 total, 8 up, 8 in 2024-03-04T16:07:39.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:38 smithi083 ceph-mon[29452]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:39.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:38 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2870534703' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2940780554"}]': finished 2024-03-04T16:07:39.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:38 smithi083 ceph-mon[24738]: osdmap e104: 8 total, 8 up, 8 in 2024-03-04T16:07:39.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:38 smithi083 ceph-mon[24738]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:41.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:41 smithi086 ceph-mon[25584]: from='client.25141 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:41.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:41 smithi086 ceph-mon[25584]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2024-03-04T16:07:41.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:41 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:41.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:41 smithi083 ceph-mon[24738]: from='client.25141 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:41.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:41 smithi083 ceph-mon[24738]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2024-03-04T16:07:41.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:41 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:41.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:41 smithi083 ceph-mon[29452]: from='client.25141 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:41.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:41 smithi083 ceph-mon[29452]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2024-03-04T16:07:41.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:41 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:42.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:42 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:07:42] "GET /metrics HTTP/1.1" 200 38131 "" "Prometheus/2.33.4" 2024-03-04T16:07:43.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:43 smithi086 ceph-mon[25584]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-04T16:07:43.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:43 smithi083 ceph-mon[24738]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-04T16:07:43.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:43 smithi083 ceph-mon[29452]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-04T16:07:45.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:45 smithi086 ceph-mon[25584]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:07:45.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:45 smithi083 ceph-mon[29452]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:07:45.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:45 smithi083 ceph-mon[24738]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:07:47.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:47 smithi086 ceph-mon[25584]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-04T16:07:47.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:47 smithi083 ceph-mon[29452]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-04T16:07:47.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:47 smithi083 ceph-mon[24738]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-04T16:07:48.984 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:48 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:07:49.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:49 smithi086 ceph-mon[25584]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-03-04T16:07:49.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:49 smithi083 ceph-mon[24738]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-03-04T16:07:49.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:49 smithi083 ceph-mon[29452]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-03-04T16:07:50.984 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:50 smithi086 systemd[1]: Stopping Ceph grafana.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:07:50.984 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:50 smithi086 bash[50832]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana.a" found: no such container 2024-03-04T16:07:50.984 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:50 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[41683]: t=2024-03-04T16:07:50+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-03-04T16:07:50.984 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:50 smithi086 podman[50842]: 2024-03-04 16:07:50.774521599 +0000 UTC m=+0.123138525 container died 9c0f7c6afe4b9a92ad01e1972af61608d582d1ceb7879bc67c97f163c8953a53 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, name=ubi8, io.buildah.version=1.24.2, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., vcs-type=git, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-04T16:07:51.418 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-mon[25584]: from='client.25141 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:51.418 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-mon[25584]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:51.418 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 podman[50842]: 2024-03-04 16:07:51.146465678 +0000 UTC m=+0.495082597 container cleanup 9c0f7c6afe4b9a92ad01e1972af61608d582d1ceb7879bc67c97f163c8953a53 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , description=Ceph Grafana Container, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, vcs-type=git) 2024-03-04T16:07:51.418 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 bash[50842]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a 2024-03-04T16:07:51.418 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 bash[50869]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana.a" found: no such container 2024-03-04T16:07:51.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:51 smithi083 ceph-mon[24738]: from='client.25141 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:51.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:51 smithi083 ceph-mon[24738]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:51.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:51 smithi083 ceph-mon[29452]: from='client.25141 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:51.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:51 smithi083 ceph-mon[29452]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:51.675 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 podman[50856]: 2024-03-04 16:07:51.418015189 +0000 UTC m=+0.638996594 container remove 9c0f7c6afe4b9a92ad01e1972af61608d582d1ceb7879bc67c97f163c8953a53 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, architecture=x86_64, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8) 2024-03-04T16:07:51.675 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-04T16:07:51.675 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 systemd[1]: Stopped Ceph grafana.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:07:51.675 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@grafana.a.service: Consumed 2.575s CPU time. 2024-03-04T16:07:51.675 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 systemd[1]: Starting Ceph grafana.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:07:51.675 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 podman[50937]: 2024-03-04 16:07:51.648843138 +0000 UTC m=+0.105901272 container create fb754d58c53f55f4e9273328632647db3e8d690e8b4a88104fd63c9b106db0d7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a, maintainer=Paul Cuzner , version=8.5, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, name=ubi8, description=Ceph Grafana Container, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64) 2024-03-04T16:07:51.675 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 podman[50937]: 2024-03-04 16:07:51.560425364 +0000 UTC m=+0.017483499 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-04T16:07:51.934 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 podman[50937]: 2024-03-04 16:07:51.892791599 +0000 UTC m=+0.349849733 container init fb754d58c53f55f4e9273328632647db3e8d690e8b4a88104fd63c9b106db0d7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, io.buildah.version=1.24.2, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, vendor=Red Hat, Inc., maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, version=8.5) 2024-03-04T16:07:51.934 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 podman[50937]: 2024-03-04 16:07:51.895481967 +0000 UTC m=+0.352540111 container start fb754d58c53f55f4e9273328632647db3e8d690e8b4a88104fd63c9b106db0d7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , io.openshift.expose-services=, release=236.1648460182, com.redhat.component=ubi8-container, vcs-type=git, distribution-scope=public, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, description=Ceph Grafana Container) 2024-03-04T16:07:51.934 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-03-04T16:07:51.935 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-03-04T16:07:51.935 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-03-04T16:07:51.935 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-03-04T16:07:51.935 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-04T16:07:51.935 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-04T16:07:51.935 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-04T16:07:51.935 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-04T16:07:51.935 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-03-04T16:07:51.935 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-03-04T16:07:51.935 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-03-04T16:07:51.935 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-03-04T16:07:51.935 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-03-04T16:07:51.935 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="App mode production" logger=settings 2024-03-04T16:07:51.935 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-03-04T16:07:51.936 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-04T16:07:51.936 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-03-04T16:07:52.224 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:07:52] "GET /metrics HTTP/1.1" 200 38131 "" "Prometheus/2.33.4" 2024-03-04T16:07:52.224 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:07:52] ENGINE Bus STOPPING 2024-03-04T16:07:52.224 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:07:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-04T16:07:52.224 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:07:52] ENGINE Bus STOPPED 2024-03-04T16:07:52.224 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:07:52] ENGINE Bus STARTING 2024-03-04T16:07:52.224 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=545.534µs 2024-03-04T16:07:52.225 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-03-04T16:07:52.225 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-03-04T16:07:52.225 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-03-04T16:07:52.225 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-03-04T16:07:52.225 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-03-04T16:07:52.225 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:51+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-03-04T16:07:52.225 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:52 smithi086 bash[50937]: fb754d58c53f55f4e9273328632647db3e8d690e8b4a88104fd63c9b106db0d7 2024-03-04T16:07:52.225 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:52 smithi086 systemd[1]: Started Ceph grafana.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:07:52.225 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:52+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2024-03-04T16:07:52.225 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:52+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-04T16:07:52.225 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:52+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2024-03-04T16:07:52.225 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:52+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-03-04T16:07:52.225 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:52+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-03-04T16:07:52.226 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:07:52+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-03-04T16:07:52.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:07:52] ENGINE Serving on http://:::9283 2024-03-04T16:07:52.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:07:52] ENGINE Bus STARTED 2024-03-04T16:07:52.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:07:52] ENGINE Bus STOPPING 2024-03-04T16:07:52.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:07:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-04T16:07:52.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:07:52] ENGINE Bus STOPPED 2024-03-04T16:07:52.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:07:52] ENGINE Bus STARTING 2024-03-04T16:07:52.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:07:52] ENGINE Serving on http://:::9283 2024-03-04T16:07:52.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:07:52] ENGINE Bus STARTED 2024-03-04T16:07:52.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:07:52] ENGINE Bus STOPPING 2024-03-04T16:07:52.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:07:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-04T16:07:52.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:07:52] ENGINE Bus STOPPED 2024-03-04T16:07:52.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:07:52] ENGINE Bus STARTING 2024-03-04T16:07:52.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:07:52] ENGINE Serving on http://:::9283 2024-03-04T16:07:52.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:07:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:07:52] ENGINE Bus STARTED 2024-03-04T16:07:53.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:53.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:53.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:07:53.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:07:53.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi083.front.sepia.ceph.com:9093"}]: dispatch 2024-03-04T16:07:53.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi083.front.sepia.ceph.com:9093"}]: dispatch 2024-03-04T16:07:53.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:53.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:07:53.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:07:53.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-03-04T16:07:53.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-03-04T16:07:53.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:53.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:07:53.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:07:53.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086.front.sepia.ceph.com:3000"}]: dispatch 2024-03-04T16:07:53.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086.front.sepia.ceph.com:3000"}]: dispatch 2024-03-04T16:07:53.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:53.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:07:53.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-04T16:07:53.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-04T16:07:53.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[29452]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:53.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:53.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:53.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:07:53.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:07:53.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi083.front.sepia.ceph.com:9093"}]: dispatch 2024-03-04T16:07:53.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi083.front.sepia.ceph.com:9093"}]: dispatch 2024-03-04T16:07:53.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:53.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:07:53.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:07:53.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-03-04T16:07:53.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-03-04T16:07:53.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:53.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:07:53.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:07:53.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086.front.sepia.ceph.com:3000"}]: dispatch 2024-03-04T16:07:53.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086.front.sepia.ceph.com:3000"}]: dispatch 2024-03-04T16:07:53.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:53.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:07:53.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-04T16:07:53.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-04T16:07:53.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:53 smithi083 ceph-mon[24738]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:53.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:53.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:53.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:07:53.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:07:53.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi083.front.sepia.ceph.com:9093"}]: dispatch 2024-03-04T16:07:53.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi083.front.sepia.ceph.com:9093"}]: dispatch 2024-03-04T16:07:53.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:53.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:07:53.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:07:53.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-03-04T16:07:53.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-03-04T16:07:53.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:53.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:07:53.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:07:53.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086.front.sepia.ceph.com:3000"}]: dispatch 2024-03-04T16:07:53.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086.front.sepia.ceph.com:3000"}]: dispatch 2024-03-04T16:07:53.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:53.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:07:53.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-04T16:07:53.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-04T16:07:53.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:53 smithi086 ceph-mon[25584]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:54.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:54 smithi083 ceph-mon[24738]: Upgrade: Updating mgr.y 2024-03-04T16:07:54.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:54 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:54.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:54 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:07:54.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:54 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:07:54.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:54 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:07:54.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:54 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:54.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:54 smithi083 ceph-mon[24738]: Deploying daemon mgr.y on smithi083 2024-03-04T16:07:54.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:54 smithi083 ceph-mon[24738]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:54.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:54 smithi083 ceph-mon[29452]: Upgrade: Updating mgr.y 2024-03-04T16:07:54.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:54 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:54.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:54 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:07:54.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:54 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:07:54.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:54 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:07:54.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:54 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:54.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:54 smithi083 ceph-mon[29452]: Deploying daemon mgr.y on smithi083 2024-03-04T16:07:54.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:54 smithi083 ceph-mon[29452]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:55.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:54 smithi086 ceph-mon[25584]: Upgrade: Updating mgr.y 2024-03-04T16:07:55.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:54 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:55.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:54 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:07:55.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:54 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:07:55.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:54 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:07:55.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:54 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:55.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:54 smithi086 ceph-mon[25584]: Deploying daemon mgr.y on smithi083 2024-03-04T16:07:55.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:54 smithi086 ceph-mon[25584]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:56.055 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:55 smithi083 systemd[1]: Stopping Ceph mgr.y for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:07:56.344 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:56 smithi083 podman[60320]: 2024-03-04 16:07:56.055273334 +0000 UTC m=+0.120025705 container died 5af09a7ff74e8e728272937bd2ee8c5ec835d53e1bf1b8d0a81e07318467092e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, GIT_CLEAN=True, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , version=8, vcs-type=git, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., release=754, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, ceph=True) 2024-03-04T16:07:56.344 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:56 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:56.344 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:56 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:56.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:56 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:56.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:56 smithi083 podman[60320]: 2024-03-04 16:07:56.517949184 +0000 UTC m=+0.582701554 container cleanup 5af09a7ff74e8e728272937bd2ee8c5ec835d53e1bf1b8d0a81e07318467092e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, architecture=x86_64, name=centos-stream, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , vcs-type=git, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, vendor=Red Hat, Inc., version=8, distribution-scope=public, build-date=2022-05-03T08:36:31.336870) 2024-03-04T16:07:56.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:56 smithi083 bash[60320]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y 2024-03-04T16:07:57.121 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:56 smithi083 podman[60334]: 2024-03-04 16:07:56.803253153 +0000 UTC m=+0.741347039 container remove 5af09a7ff74e8e728272937bd2ee8c5ec835d53e1bf1b8d0a81e07318467092e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, name=centos-stream, vcs-type=git, GIT_BRANCH=HEAD, GIT_CLEAN=True, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , ceph=True, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, release=754, vendor=Red Hat, Inc., RELEASE=HEAD, version=8) 2024-03-04T16:07:57.121 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:56 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-04T16:07:57.121 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:56 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-03-04T16:07:57.121 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:56 smithi083 systemd[1]: Stopped Ceph mgr.y for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:07:57.121 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:56 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mgr.y.service: Consumed 52.820s CPU time. 2024-03-04T16:07:57.121 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:57 smithi083 systemd[1]: Starting Ceph mgr.y for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:07:57.452 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:57 smithi083 ceph-mon[24738]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:57.453 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:57 smithi083 ceph-mon[29452]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:57.453 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:57 smithi083 podman[60468]: 2024-03-04 16:07:57.241083325 +0000 UTC m=+0.018040075 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:07:57.454 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:57 smithi083 podman[60468]: 2024-03-04 16:07:57.346152925 +0000 UTC m=+0.123109672 container create 164dd93609ee68c6b3736fef7f15373aa062ae504407c5f92e7f8282295a5cd2 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:07:57.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:57 smithi086 ceph-mon[25584]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:07:57.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:57 smithi083 podman[60468]: 2024-03-04 16:07:57.557316946 +0000 UTC m=+0.334273704 container init 164dd93609ee68c6b3736fef7f15373aa062ae504407c5f92e7f8282295a5cd2 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-04T16:07:57.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:57 smithi083 podman[60468]: 2024-03-04 16:07:57.559925409 +0000 UTC m=+0.336882159 container start 164dd93609ee68c6b3736fef7f15373aa062ae504407c5f92e7f8282295a5cd2 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:07:57.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:57 smithi083 ceph-mgr[60488]: -- 172.21.15.83:0/1882276610 <== mon.0 v2:172.21.15.83:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5622d88945a0 con 0x5622d8868c00 2024-03-04T16:07:57.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:57 smithi083 bash[60468]: 164dd93609ee68c6b3736fef7f15373aa062ae504407c5f92e7f8282295a5cd2 2024-03-04T16:07:57.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:57 smithi083 systemd[1]: Started Ceph mgr.y for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:07:58.488 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:58 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:07:58.129+0000 7fd6ce5db200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T16:07:58.488 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:58 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:07:58.237+0000 7fd6ce5db200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T16:07:58.921 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:58 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:58.921 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:58 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:58.921 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:58 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:58.921 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:58 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:58.922 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:07:58 smithi083 ceph-mon[29452]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:58.922 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:58 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:58.922 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:58 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:58.922 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:58 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:58.922 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:58 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:58.922 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:07:58 smithi083 ceph-mon[24738]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:58.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:58 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:58.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:58 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:58.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:58 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:58.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:58 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:07:58.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:07:58 smithi086 ceph-mon[25584]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:07:58.985 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:07:58 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:07:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:07:59.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:58 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:07:58.920+0000 7fd6ce5db200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T16:07:59.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:59 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:07:59.014+0000 7fd6ce5db200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T16:07:59.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:59 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:07:59.165+0000 7fd6ce5db200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T16:07:59.579 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:59 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:07:59.261+0000 7fd6ce5db200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T16:07:59.579 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:59 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:07:59.347+0000 7fd6ce5db200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T16:07:59.843 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:59 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:07:59.620+0000 7fd6ce5db200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T16:07:59.843 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:59 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:07:59.745+0000 7fd6ce5db200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T16:07:59.843 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:59 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:07:59.842+0000 7fd6ce5db200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T16:08:00.140 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:07:59 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:07:59.953+0000 7fd6ce5db200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T16:08:00.488 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:00 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:00.139+0000 7fd6ce5db200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T16:08:01.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:01 smithi086 ceph-mon[25584]: from='client.25141 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:01.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:01 smithi086 ceph-mon[25584]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:01.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:01 smithi083 ceph-mon[29452]: from='client.25141 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:01.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:01 smithi083 ceph-mon[29452]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:01.490 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:01 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:01.176+0000 7fd6ce5db200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T16:08:01.490 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:01 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:01.310+0000 7fd6ce5db200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T16:08:01.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:01 smithi083 ceph-mon[24738]: from='client.25141 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:01.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:01 smithi083 ceph-mon[24738]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:01.978 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:01 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:01.673+0000 7fd6ce5db200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T16:08:01.978 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:01 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:01.771+0000 7fd6ce5db200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T16:08:02.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:01 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:01.977+0000 7fd6ce5db200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:08:02.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:02.167+0000 7fd6ce5db200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T16:08:02.340 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:02 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:08:02] "GET /metrics HTTP/1.1" 200 38138 "" "Prometheus/2.33.4" 2024-03-04T16:08:02.652 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:08:02.832 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:02.538+0000 7fd6ce5db200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:08:03.109 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:02 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:02.832+0000 7fd6ce5db200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T16:08:03.332 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:08:03.332 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (49s) 1s ago 5m 14.8M - 0.23.0 ba2b418f427c 3c6dc3779a0e 2024-03-04T16:08:03.332 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 starting - - - - 2024-03-04T16:08:03.333 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 running (34s) 1s ago 4m 47.8M - 3.5 e1d6a67b021e 2b78519ae16b 2024-03-04T16:08:03.333 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443,9283 running (92s) 69s ago 10m 490M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e a73348a7968e 2024-03-04T16:08:03.333 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:8443,9283,8765 running (6s) 1s ago 11m 66.8M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 164dd93609ee 2024-03-04T16:08:03.333 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (11m) 1s ago 12m 52.7M 2048M 17.2.0 e1d6a67b021e d2ba878ed364 2024-03-04T16:08:03.333 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (10m) 69s ago 10m 39.6M 2048M 17.2.0 e1d6a67b021e d1b972ebdd2a 2024-03-04T16:08:03.333 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (10m) 1s ago 10m 42.6M 2048M 17.2.0 e1d6a67b021e e74b0ab73a80 2024-03-04T16:08:03.333 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (60s) 1s ago 6m 9248k - 1.3.1 1dbe0e931976 9bf54b460640 2024-03-04T16:08:03.333 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 starting - - - - 2024-03-04T16:08:03.333 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (10m) 1s ago 10m 51.9M 2736M 17.2.0 e1d6a67b021e 56679a9f787b 2024-03-04T16:08:03.333 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (9m) 1s ago 9m 54.8M 2736M 17.2.0 e1d6a67b021e c1070c1a315a 2024-03-04T16:08:03.333 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (9m) 1s ago 9m 48.0M 2736M 17.2.0 e1d6a67b021e 2b913c19f581 2024-03-04T16:08:03.333 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (8m) 1s ago 8m 50.1M 2736M 17.2.0 e1d6a67b021e 9f9a4e2ec5da 2024-03-04T16:08:03.333 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (8m) 69s ago 8m 51.5M 2992M 17.2.0 e1d6a67b021e b3a97f83f1c7 2024-03-04T16:08:03.334 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (8m) 69s ago 8m 49.6M 2992M 17.2.0 e1d6a67b021e b82ca42b7353 2024-03-04T16:08:03.334 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (7m) 69s ago 7m 48.0M 2992M 17.2.0 e1d6a67b021e 18cedf14321d 2024-03-04T16:08:03.334 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (7m) 69s ago 7m 52.2M 2992M 17.2.0 e1d6a67b021e 390e1798c738 2024-03-04T16:08:03.334 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 starting - - - - 2024-03-04T16:08:03.334 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (5m) 1s ago 5m 72.2M - 17.2.0 e1d6a67b021e 65f14042f354 2024-03-04T16:08:03.334 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (5m) 69s ago 5m 73.0M - 17.2.0 e1d6a67b021e 78175cc6dc5d 2024-03-04T16:08:03.334 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (4m) 1s ago 4m 70.5M - 17.2.0 e1d6a67b021e 01dcf858bd62 2024-03-04T16:08:03.334 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (4m) 69s ago 4m 75.0M - 17.2.0 e1d6a67b021e 8f70a5ce9f0d 2024-03-04T16:08:03.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:03 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:03.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:03 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:03.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:03 smithi086 ceph-mon[25584]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:08:03.488 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:03 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:03.108+0000 7fd6ce5db200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T16:08:03.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:03 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:03.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:03 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:03.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:03 smithi083 ceph-mon[24738]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:08:03.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:03 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:03.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:03 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:03.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:03 smithi083 ceph-mon[29452]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:08:03.752 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:08:03.752 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:08:03.752 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-04T16:08:03.752 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:08:03.752 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:08:03.752 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-04T16:08:03.752 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 1 2024-03-04T16:08:03.752 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:08:03.752 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:08:03.752 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-04T16:08:03.752 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:08:03.752 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": {}, 2024-03-04T16:08:03.753 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:08:03.753 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-04T16:08:03.753 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:08:03.753 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:08:03.753 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-03-04T16:08:03.753 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 1 2024-03-04T16:08:03.753 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:08:03.753 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:08:04.107 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:08:04.107 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:08:04.107 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:08:04.107 INFO:teuthology.orchestra.run.smithi083.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:08:04.107 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [ 2024-03-04T16:08:04.107 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr" 2024-03-04T16:08:04.107 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-04T16:08:04.107 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "2/23 daemons upgraded", 2024-03-04T16:08:04.107 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "Currently upgrading mgr daemons", 2024-03-04T16:08:04.108 INFO:teuthology.orchestra.run.smithi083.stdout: "is_paused": false 2024-03-04T16:08:04.108 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:08:04.305 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:04 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:04.017+0000 7fd6ce5db200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T16:08:04.526 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_OK 2024-03-04T16:08:04.620 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:04 smithi083 ceph-mon[24738]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:04.620 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:04 smithi083 ceph-mon[24738]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:04.621 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:04 smithi083 ceph-mon[24738]: from='client.25249 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:04.621 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:04 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:04.621 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:04 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:04.621 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:04 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/344451046' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:04.621 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:04 smithi083 ceph-mon[29452]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:04.621 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:04 smithi083 ceph-mon[29452]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:04.621 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:04 smithi083 ceph-mon[29452]: from='client.25249 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:04.621 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:04 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:04.621 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:04 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:04.621 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:04 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/344451046' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:04.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:04 smithi086 ceph-mon[25584]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:04.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:04 smithi086 ceph-mon[25584]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:04.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:04 smithi086 ceph-mon[25584]: from='client.25249 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:04.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:04 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:04.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:04 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:04.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:04 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/344451046' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:04.946 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:04 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:04.619+0000 7fd6ce5db200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T16:08:04.947 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:04 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:04.722+0000 7fd6ce5db200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T16:08:04.947 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:04 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:04.946+0000 7fd6ce5db200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T16:08:05.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:04 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: [04/Mar/2024:16:08:04] ENGINE Bus STARTING 2024-03-04T16:08:05.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: CherryPy Checker: 2024-03-04T16:08:05.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: The Application mounted at '' has an empty config. 2024-03-04T16:08:05.239 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:05.239 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: [04/Mar/2024:16:08:05] ENGINE Serving on http://:::9283 2024-03-04T16:08:05.239 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: [04/Mar/2024:16:08:05] ENGINE Bus STARTED 2024-03-04T16:08:05.538 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-mon[29452]: from='client.25261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:05.538 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-mon[29452]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:05.538 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1635662845' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:08:05.538 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-mon[29452]: Standby manager daemon y restarted 2024-03-04T16:08:05.538 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-mon[29452]: Standby manager daemon y started 2024-03-04T16:08:05.539 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-04T16:08:05.539 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:08:05.539 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-04T16:08:05.539 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:08:05.539 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-mon[24738]: from='client.25261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:05.539 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-mon[24738]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:05.539 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1635662845' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:08:05.539 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-mon[24738]: Standby manager daemon y restarted 2024-03-04T16:08:05.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-mon[24738]: Standby manager daemon y started 2024-03-04T16:08:05.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-04T16:08:05.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:08:05.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-04T16:08:05.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:05 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:08:05.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:05 smithi086 ceph-mon[25584]: from='client.25261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:05.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:05 smithi086 ceph-mon[25584]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:05.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:05 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1635662845' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:08:05.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:05 smithi086 ceph-mon[25584]: Standby manager daemon y restarted 2024-03-04T16:08:05.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:05 smithi086 ceph-mon[25584]: Standby manager daemon y started 2024-03-04T16:08:05.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:05 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-04T16:08:05.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:05 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:08:05.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:05 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-04T16:08:05.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:05 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:08:06.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:06 smithi086 ceph-mon[25584]: mgrmap e26: x(active, since 86s), standbys: y 2024-03-04T16:08:06.970 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:06 smithi083 ceph-mon[29452]: mgrmap e26: x(active, since 86s), standbys: y 2024-03-04T16:08:06.970 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:06 smithi083 ceph-mon[24738]: mgrmap e26: x(active, since 86s), standbys: y 2024-03-04T16:08:07.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:07 smithi086 ceph-mon[25584]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:08:07.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:07 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:07.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:07 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:07.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:07 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:07.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:07 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:07.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:07 smithi083 ceph-mon[24738]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:08:07.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:07 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:07.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:07 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:07.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:07 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:07.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:07 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:07.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:07 smithi083 ceph-mon[29452]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:08:07.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:07 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:07.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:07 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:07.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:07 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:07.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:07 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:08.799 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:08 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:08.799 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:08 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:08:08.799 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:08 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:08.799 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:08 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:08:08.799 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:08 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:08:08.799 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:08 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:08.799 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:08 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:08.799 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:08 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:08:08.800 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:08 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:08.800 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:08 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:08:08.800 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:08 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:08:08.800 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:08 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:08.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:08 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:08.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:08 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:08:08.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:08 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' 2024-03-04T16:08:08.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:08 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:08:08.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:08 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:08:08.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:08 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:08.985 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:08 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:08:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:08:09.757 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:09 smithi083 ceph-mon[29452]: Reconfiguring iscsi.foo.smithi083.fnypow (dependencies changed)... 2024-03-04T16:08:09.757 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:09 smithi083 ceph-mon[29452]: Reconfiguring daemon iscsi.foo.smithi083.fnypow on smithi083 2024-03-04T16:08:09.757 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:09 smithi083 ceph-mon[29452]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:09.757 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:09 smithi083 ceph-mon[24738]: Reconfiguring iscsi.foo.smithi083.fnypow (dependencies changed)... 2024-03-04T16:08:09.758 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:09 smithi083 ceph-mon[24738]: Reconfiguring daemon iscsi.foo.smithi083.fnypow on smithi083 2024-03-04T16:08:09.758 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:09 smithi083 ceph-mon[24738]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:09.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:09 smithi086 ceph-mon[25584]: Reconfiguring iscsi.foo.smithi083.fnypow (dependencies changed)... 2024-03-04T16:08:09.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:09 smithi086 ceph-mon[25584]: Reconfiguring daemon iscsi.foo.smithi083.fnypow on smithi083 2024-03-04T16:08:09.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:09 smithi086 ceph-mon[25584]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:11.196 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:11 smithi083 ceph-mon[24738]: from='client.25141 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:11.196 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:11 smithi083 ceph-mon[24738]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:11.196 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:11 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:11.197 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:11 smithi083 ceph-mon[29452]: from='client.25141 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:11.197 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:11 smithi083 ceph-mon[29452]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:11.197 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:11 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:11.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-mon[25584]: from='client.25141 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:11.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-mon[25584]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:11.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:11.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:11.269+0000 7fc74f87a700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi083.fnypow ... 2024-03-04T16:08:11.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Creating ceph-iscsi config... 2024-03-04T16:08:11.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/iscsi-gateway.cfg 2024-03-04T16:08:11.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/tcmu-runner-entrypoint.sh 2024-03-04T16:08:11.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Non-zero exit code 1 from systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow 2024-03-04T16:08:11.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: systemctl: stderr Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:08:11.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: systemctl: stderr See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:08:11.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Traceback (most recent call last): 2024-03-04T16:08:11.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-04T16:08:11.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: return _run_code(code, main_globals, None, 2024-03-04T16:08:11.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-04T16:08:11.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: exec(code, run_globals) 2024-03-04T16:08:11.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5527, in 2024-03-04T16:08:11.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5515, in main 2024-03-04T16:08:11.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3043, in command_deploy_from 2024-03-04T16:08:11.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3078, in _common_deploy 2024-03-04T16:08:11.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3098, in _deploy_daemon_container 2024-03-04T16:08:11.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 1070, in deploy_daemon 2024-03-04T16:08:11.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-04T16:08:11.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: RuntimeError: Failed command: systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow: Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:08:11.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:08:11.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Traceback (most recent call last): 2024-03-04T16:08:11.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-04T16:08:11.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-04T16:08:11.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2382, in _daemon_action 2024-03-04T16:08:11.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-04T16:08:11.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-04T16:08:11.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: return self.event_loop.get_result(coro, timeout) 2024-03-04T16:08:11.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-04T16:08:11.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: return future.result(timeout) 2024-03-04T16:08:11.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-04T16:08:11.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: return self.__get_result() 2024-03-04T16:08:11.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-04T16:08:11.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: raise self._exception 2024-03-04T16:08:11.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-04T16:08:11.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: config_blobs=daemon_spec.final_config, 2024-03-04T16:08:11.487 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-04T16:08:11.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-04T16:08:11.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi083.fnypow ... 2024-03-04T16:08:11.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Creating ceph-iscsi config... 2024-03-04T16:08:11.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/iscsi-gateway.cfg 2024-03-04T16:08:11.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/tcmu-runner-entrypoint.sh 2024-03-04T16:08:11.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Non-zero exit code 1 from systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow 2024-03-04T16:08:11.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: systemctl: stderr Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:08:11.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: systemctl: stderr See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:08:11.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: Traceback (most recent call last): 2024-03-04T16:08:11.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-04T16:08:11.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: return _run_code(code, main_globals, None, 2024-03-04T16:08:11.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-04T16:08:11.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: exec(code, run_globals) 2024-03-04T16:08:11.488 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5527, in 2024-03-04T16:08:11.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5515, in main 2024-03-04T16:08:11.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3043, in command_deploy_from 2024-03-04T16:08:11.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3078, in _common_deploy 2024-03-04T16:08:11.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3098, in _deploy_daemon_container 2024-03-04T16:08:11.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 1070, in deploy_daemon 2024-03-04T16:08:11.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-04T16:08:11.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: RuntimeError: Failed command: systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow: Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:08:11.489 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:08:12.362 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:12 smithi086 ceph-mon[25584]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi083.fnypow ... 2024-03-04T16:08:12.362 INFO:journalctl@ceph.mon.b.smithi086.stdout: Creating ceph-iscsi config... 2024-03-04T16:08:12.362 INFO:journalctl@ceph.mon.b.smithi086.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/iscsi-gateway.cfg 2024-03-04T16:08:12.362 INFO:journalctl@ceph.mon.b.smithi086.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/tcmu-runner-entrypoint.sh 2024-03-04T16:08:12.362 INFO:journalctl@ceph.mon.b.smithi086.stdout: Non-zero exit code 1 from systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow 2024-03-04T16:08:12.362 INFO:journalctl@ceph.mon.b.smithi086.stdout: systemctl: stderr Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:08:12.362 INFO:journalctl@ceph.mon.b.smithi086.stdout: systemctl: stderr See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:08:12.363 INFO:journalctl@ceph.mon.b.smithi086.stdout: Traceback (most recent call last): 2024-03-04T16:08:12.363 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-04T16:08:12.363 INFO:journalctl@ceph.mon.b.smithi086.stdout: return _run_code(code, main_globals, None, 2024-03-04T16:08:12.363 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-04T16:08:12.363 INFO:journalctl@ceph.mon.b.smithi086.stdout: exec(code, run_globals) 2024-03-04T16:08:12.363 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5527, in 2024-03-04T16:08:12.363 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5515, in main 2024-03-04T16:08:12.363 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3043, in command_deploy_from 2024-03-04T16:08:12.363 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3078, in _common_deploy 2024-03-04T16:08:12.363 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3098, in _deploy_daemon_container 2024-03-04T16:08:12.363 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 1070, in deploy_daemon 2024-03-04T16:08:12.363 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-04T16:08:12.364 INFO:journalctl@ceph.mon.b.smithi086.stdout: RuntimeError: Failed command: systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow: Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:08:12.364 INFO:journalctl@ceph.mon.b.smithi086.stdout: See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:08:12.364 INFO:journalctl@ceph.mon.b.smithi086.stdout: Traceback (most recent call last): 2024-03-04T16:08:12.364 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-04T16:08:12.364 INFO:journalctl@ceph.mon.b.smithi086.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-04T16:08:12.364 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2382, in _daemon_action 2024-03-04T16:08:12.364 INFO:journalctl@ceph.mon.b.smithi086.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-04T16:08:12.364 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-04T16:08:12.364 INFO:journalctl@ceph.mon.b.smithi086.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-04T16:08:12.364 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-04T16:08:12.364 INFO:journalctl@ceph.mon.b.smithi086.stdout: return future.result(timeout) 2024-03-04T16:08:12.364 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-04T16:08:12.364 INFO:journalctl@ceph.mon.b.smithi086.stdout: return self.__get_result() 2024-03-04T16:08:12.365 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-04T16:08:12.365 INFO:journalctl@ceph.mon.b.smithi086.stdout: raise self._exception 2024-03-04T16:08:12.365 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-04T16:08:12.365 INFO:journalctl@ceph.mon.b.smithi086.stdout: config_blobs=daemon_spec.final_config, 2024-03-04T16:08:12.365 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-04T16:08:12.365 INFO:journalctl@ceph.mon.b.smithi086.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-04T16:08:12.365 INFO:journalctl@ceph.mon.b.smithi086.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi083.fnypow ... 2024-03-04T16:08:12.365 INFO:journalctl@ceph.mon.b.smithi086.stdout: Creating ceph-iscsi config... 2024-03-04T16:08:12.365 INFO:journalctl@ceph.mon.b.smithi086.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/iscsi-gateway.cfg 2024-03-04T16:08:12.365 INFO:journalctl@ceph.mon.b.smithi086.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/tcmu-runner-entrypoint.sh 2024-03-04T16:08:12.365 INFO:journalctl@ceph.mon.b.smithi086.stdout: Non-zero exit code 1 from systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow 2024-03-04T16:08:12.365 INFO:journalctl@ceph.mon.b.smithi086.stdout: systemctl: stderr Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:08:12.365 INFO:journalctl@ceph.mon.b.smithi086.stdout: systemctl: stderr See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:08:12.365 INFO:journalctl@ceph.mon.b.smithi086.stdout: Traceback (most recent call last): 2024-03-04T16:08:12.365 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-04T16:08:12.365 INFO:journalctl@ceph.mon.b.smithi086.stdout: return _run_code(code, main_globals, None, 2024-03-04T16:08:12.366 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-04T16:08:12.366 INFO:journalctl@ceph.mon.b.smithi086.stdout: exec(code, run_globals) 2024-03-04T16:08:12.366 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5527, in 2024-03-04T16:08:12.366 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5515, in main 2024-03-04T16:08:12.366 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3043, in command_deploy_from 2024-03-04T16:08:12.366 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3078, in _common_deploy 2024-03-04T16:08:12.366 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:12 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:08:12] "GET /metrics HTTP/1.1" 200 38134 "" "Prometheus/2.33.4" 2024-03-04T16:08:12.366 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:12 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:12.202+0000 7fc7bd147700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-04T16:08:12.366 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:12 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: ignoring --setuser ceph since I am not root 2024-03-04T16:08:12.366 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:12 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: ignoring --setgroup ceph since I am not root 2024-03-04T16:08:12.367 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:12 smithi086 ceph-mgr[46779]: -- 172.21.15.86:0/4224603119 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ffffcbe5a0 con 0x55ffffc93000 2024-03-04T16:08:12.367 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3098, in _deploy_daemon_container 2024-03-04T16:08:12.367 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 1070, in deploy_daemon 2024-03-04T16:08:12.367 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-04T16:08:12.367 INFO:journalctl@ceph.mon.b.smithi086.stdout: RuntimeError: Failed command: systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow: Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:08:12.367 INFO:journalctl@ceph.mon.b.smithi086.stdout: See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:08:12.367 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:12 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:12.367 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:12 smithi086 ceph-mon[25584]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-04T16:08:12.367 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:12 smithi086 ceph-mon[25584]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-04T16:08:12.367 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:12 smithi086 ceph-mon[25584]: Failing over to other MGR 2024-03-04T16:08:12.367 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:12 smithi086 ceph-mon[25584]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-04T16:08:12.368 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:12 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-04T16:08:12.368 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:12 smithi086 ceph-mon[25584]: osdmap e105: 8 total, 8 up, 8 in 2024-03-04T16:08:12.448 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-mon[24738]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi083.fnypow ... 2024-03-04T16:08:12.448 INFO:journalctl@ceph.mon.a.smithi083.stdout: Creating ceph-iscsi config... 2024-03-04T16:08:12.448 INFO:journalctl@ceph.mon.a.smithi083.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/iscsi-gateway.cfg 2024-03-04T16:08:12.448 INFO:journalctl@ceph.mon.a.smithi083.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/tcmu-runner-entrypoint.sh 2024-03-04T16:08:12.448 INFO:journalctl@ceph.mon.a.smithi083.stdout: Non-zero exit code 1 from systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow 2024-03-04T16:08:12.448 INFO:journalctl@ceph.mon.a.smithi083.stdout: systemctl: stderr Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:08:12.449 INFO:journalctl@ceph.mon.a.smithi083.stdout: systemctl: stderr See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:08:12.449 INFO:journalctl@ceph.mon.a.smithi083.stdout: Traceback (most recent call last): 2024-03-04T16:08:12.449 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-04T16:08:12.449 INFO:journalctl@ceph.mon.a.smithi083.stdout: return _run_code(code, main_globals, None, 2024-03-04T16:08:12.449 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-04T16:08:12.449 INFO:journalctl@ceph.mon.a.smithi083.stdout: exec(code, run_globals) 2024-03-04T16:08:12.449 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5527, in 2024-03-04T16:08:12.449 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5515, in main 2024-03-04T16:08:12.449 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3043, in command_deploy_from 2024-03-04T16:08:12.449 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3078, in _common_deploy 2024-03-04T16:08:12.449 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3098, in _deploy_daemon_container 2024-03-04T16:08:12.449 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 1070, in deploy_daemon 2024-03-04T16:08:12.449 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-04T16:08:12.449 INFO:journalctl@ceph.mon.a.smithi083.stdout: RuntimeError: Failed command: systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow: Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:08:12.450 INFO:journalctl@ceph.mon.a.smithi083.stdout: See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:08:12.450 INFO:journalctl@ceph.mon.a.smithi083.stdout: Traceback (most recent call last): 2024-03-04T16:08:12.450 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-04T16:08:12.450 INFO:journalctl@ceph.mon.a.smithi083.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-04T16:08:12.450 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2382, in _daemon_action 2024-03-04T16:08:12.450 INFO:journalctl@ceph.mon.a.smithi083.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-04T16:08:12.450 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-04T16:08:12.450 INFO:journalctl@ceph.mon.a.smithi083.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-04T16:08:12.450 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-04T16:08:12.450 INFO:journalctl@ceph.mon.a.smithi083.stdout: return future.result(timeout) 2024-03-04T16:08:12.450 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-04T16:08:12.450 INFO:journalctl@ceph.mon.a.smithi083.stdout: return self.__get_result() 2024-03-04T16:08:12.450 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-04T16:08:12.450 INFO:journalctl@ceph.mon.a.smithi083.stdout: raise self._exception 2024-03-04T16:08:12.451 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-04T16:08:12.451 INFO:journalctl@ceph.mon.a.smithi083.stdout: config_blobs=daemon_spec.final_config, 2024-03-04T16:08:12.451 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-04T16:08:12.451 INFO:journalctl@ceph.mon.a.smithi083.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-04T16:08:12.451 INFO:journalctl@ceph.mon.a.smithi083.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi083.fnypow ... 2024-03-04T16:08:12.451 INFO:journalctl@ceph.mon.a.smithi083.stdout: Creating ceph-iscsi config... 2024-03-04T16:08:12.451 INFO:journalctl@ceph.mon.a.smithi083.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/iscsi-gateway.cfg 2024-03-04T16:08:12.451 INFO:journalctl@ceph.mon.a.smithi083.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/tcmu-runner-entrypoint.sh 2024-03-04T16:08:12.451 INFO:journalctl@ceph.mon.a.smithi083.stdout: Non-zero exit code 1 from systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow 2024-03-04T16:08:12.451 INFO:journalctl@ceph.mon.a.smithi083.stdout: systemctl: stderr Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:08:12.451 INFO:journalctl@ceph.mon.a.smithi083.stdout: systemctl: stderr See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:08:12.451 INFO:journalctl@ceph.mon.a.smithi083.stdout: Traceback (most recent call last): 2024-03-04T16:08:12.451 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-04T16:08:12.451 INFO:journalctl@ceph.mon.a.smithi083.stdout: return _run_code(code, main_globals, None, 2024-03-04T16:08:12.452 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-04T16:08:12.452 INFO:journalctl@ceph.mon.a.smithi083.stdout: exec(code, run_globals) 2024-03-04T16:08:12.452 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5527, in 2024-03-04T16:08:12.452 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5515, in main 2024-03-04T16:08:12.452 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3043, in command_deploy_from 2024-03-04T16:08:12.452 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3078, in _common_deploy 2024-03-04T16:08:12.452 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: [04/Mar/2024:16:08:12] ENGINE Bus STOPPING 2024-03-04T16:08:12.452 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: [04/Mar/2024:16:08:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-04T16:08:12.452 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: [04/Mar/2024:16:08:12] ENGINE Bus STOPPED 2024-03-04T16:08:12.452 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: [04/Mar/2024:16:08:12] ENGINE Bus STARTING 2024-03-04T16:08:12.453 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: [04/Mar/2024:16:08:12] ENGINE Serving on http://:::9283 2024-03-04T16:08:12.453 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: [04/Mar/2024:16:08:12] ENGINE Bus STARTED 2024-03-04T16:08:12.453 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3098, in _deploy_daemon_container 2024-03-04T16:08:12.453 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 1070, in deploy_daemon 2024-03-04T16:08:12.453 INFO:journalctl@ceph.mon.a.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-04T16:08:12.453 INFO:journalctl@ceph.mon.a.smithi083.stdout: RuntimeError: Failed command: systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow: Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:08:12.453 INFO:journalctl@ceph.mon.a.smithi083.stdout: See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:08:12.453 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:12.453 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-mon[24738]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-04T16:08:12.453 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-mon[24738]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-04T16:08:12.454 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-mon[24738]: Failing over to other MGR 2024-03-04T16:08:12.454 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-mon[24738]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-04T16:08:12.454 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-04T16:08:12.454 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-mon[24738]: osdmap e105: 8 total, 8 up, 8 in 2024-03-04T16:08:12.454 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-mon[29452]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi083.fnypow ... 2024-03-04T16:08:12.454 INFO:journalctl@ceph.mon.c.smithi083.stdout: Creating ceph-iscsi config... 2024-03-04T16:08:12.454 INFO:journalctl@ceph.mon.c.smithi083.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/iscsi-gateway.cfg 2024-03-04T16:08:12.455 INFO:journalctl@ceph.mon.c.smithi083.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/tcmu-runner-entrypoint.sh 2024-03-04T16:08:12.455 INFO:journalctl@ceph.mon.c.smithi083.stdout: Non-zero exit code 1 from systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow 2024-03-04T16:08:12.455 INFO:journalctl@ceph.mon.c.smithi083.stdout: systemctl: stderr Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:08:12.455 INFO:journalctl@ceph.mon.c.smithi083.stdout: systemctl: stderr See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:08:12.455 INFO:journalctl@ceph.mon.c.smithi083.stdout: Traceback (most recent call last): 2024-03-04T16:08:12.455 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-04T16:08:12.455 INFO:journalctl@ceph.mon.c.smithi083.stdout: return _run_code(code, main_globals, None, 2024-03-04T16:08:12.455 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-04T16:08:12.455 INFO:journalctl@ceph.mon.c.smithi083.stdout: exec(code, run_globals) 2024-03-04T16:08:12.455 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5527, in 2024-03-04T16:08:12.455 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5515, in main 2024-03-04T16:08:12.455 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3043, in command_deploy_from 2024-03-04T16:08:12.455 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3078, in _common_deploy 2024-03-04T16:08:12.455 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3098, in _deploy_daemon_container 2024-03-04T16:08:12.456 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 1070, in deploy_daemon 2024-03-04T16:08:12.456 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-04T16:08:12.456 INFO:journalctl@ceph.mon.c.smithi083.stdout: RuntimeError: Failed command: systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow: Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:08:12.456 INFO:journalctl@ceph.mon.c.smithi083.stdout: See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:08:12.456 INFO:journalctl@ceph.mon.c.smithi083.stdout: Traceback (most recent call last): 2024-03-04T16:08:12.456 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-04T16:08:12.456 INFO:journalctl@ceph.mon.c.smithi083.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-04T16:08:12.456 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2382, in _daemon_action 2024-03-04T16:08:12.456 INFO:journalctl@ceph.mon.c.smithi083.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-04T16:08:12.456 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-04T16:08:12.456 INFO:journalctl@ceph.mon.c.smithi083.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-04T16:08:12.456 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-04T16:08:12.456 INFO:journalctl@ceph.mon.c.smithi083.stdout: return future.result(timeout) 2024-03-04T16:08:12.457 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-04T16:08:12.457 INFO:journalctl@ceph.mon.c.smithi083.stdout: return self.__get_result() 2024-03-04T16:08:12.457 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-04T16:08:12.457 INFO:journalctl@ceph.mon.c.smithi083.stdout: raise self._exception 2024-03-04T16:08:12.457 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-04T16:08:12.457 INFO:journalctl@ceph.mon.c.smithi083.stdout: config_blobs=daemon_spec.final_config, 2024-03-04T16:08:12.457 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-04T16:08:12.457 INFO:journalctl@ceph.mon.c.smithi083.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-04T16:08:12.457 INFO:journalctl@ceph.mon.c.smithi083.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi083.fnypow ... 2024-03-04T16:08:12.457 INFO:journalctl@ceph.mon.c.smithi083.stdout: Creating ceph-iscsi config... 2024-03-04T16:08:12.457 INFO:journalctl@ceph.mon.c.smithi083.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/iscsi-gateway.cfg 2024-03-04T16:08:12.457 INFO:journalctl@ceph.mon.c.smithi083.stdout: Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/tcmu-runner-entrypoint.sh 2024-03-04T16:08:12.457 INFO:journalctl@ceph.mon.c.smithi083.stdout: Non-zero exit code 1 from systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow 2024-03-04T16:08:12.457 INFO:journalctl@ceph.mon.c.smithi083.stdout: systemctl: stderr Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:08:12.457 INFO:journalctl@ceph.mon.c.smithi083.stdout: systemctl: stderr See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:08:12.458 INFO:journalctl@ceph.mon.c.smithi083.stdout: Traceback (most recent call last): 2024-03-04T16:08:12.458 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-04T16:08:12.458 INFO:journalctl@ceph.mon.c.smithi083.stdout: return _run_code(code, main_globals, None, 2024-03-04T16:08:12.458 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-04T16:08:12.458 INFO:journalctl@ceph.mon.c.smithi083.stdout: exec(code, run_globals) 2024-03-04T16:08:12.458 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5527, in 2024-03-04T16:08:12.458 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5515, in main 2024-03-04T16:08:12.458 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3043, in command_deploy_from 2024-03-04T16:08:12.458 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3078, in _common_deploy 2024-03-04T16:08:12.458 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3098, in _deploy_daemon_container 2024-03-04T16:08:12.458 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 1070, in deploy_daemon 2024-03-04T16:08:12.458 INFO:journalctl@ceph.mon.c.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-04T16:08:12.458 INFO:journalctl@ceph.mon.c.smithi083.stdout: RuntimeError: Failed command: systemctl restart ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow: Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:08:12.458 INFO:journalctl@ceph.mon.c.smithi083.stdout: See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:08:12.459 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:12.459 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-mon[29452]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-04T16:08:12.459 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-mon[29452]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-04T16:08:12.459 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-mon[29452]: Failing over to other MGR 2024-03-04T16:08:12.459 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-mon[29452]: from='mgr.34898 172.21.15.86:0/567568066' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-04T16:08:12.459 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-04T16:08:12.459 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:12 smithi083 ceph-mon[29452]: osdmap e105: 8 total, 8 up, 8 in 2024-03-04T16:08:13.212 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:12 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:12.778+0000 7fea84f0c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T16:08:13.212 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:12 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:12.874+0000 7fea84f0c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T16:08:13.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.34898 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-04T16:08:13.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: mgrmap e27: y(active, starting, since 0.919825s) 2024-03-04T16:08:13.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:08:13.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:08:13.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:08:13.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T16:08:13.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:08:13.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:08:13.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:08:13.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:08:13.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:08:13.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:08:13.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:08:13.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:08:13.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:08:13.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:08:13.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:08:13.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: Manager daemon y is now available 2024-03-04T16:08:13.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:13.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:13.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:08:13.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:08:13.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:08:13.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:08:13.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.34898 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-04T16:08:13.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: mgrmap e27: y(active, starting, since 0.919825s) 2024-03-04T16:08:13.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:08:13.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:08:13.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:08:13.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T16:08:13.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:08:13.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:08:13.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:08:13.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:08:13.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:08:13.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:08:13.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:08:13.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:08:13.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:08:13.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:08:13.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:08:13.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: Manager daemon y is now available 2024-03-04T16:08:13.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:13.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:13.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:08:13.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:08:13.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:08:13.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:08:13.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.34898 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-04T16:08:13.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: mgrmap e27: y(active, starting, since 0.919825s) 2024-03-04T16:08:13.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:08:13.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:08:13.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:08:13.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T16:08:13.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:08:13.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:08:13.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:08:13.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:08:13.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:08:13.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:08:13.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:08:13.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:08:13.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:08:13.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:08:13.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:08:13.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: Manager daemon y is now available 2024-03-04T16:08:13.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:13.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:13.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:08:13.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:08:13.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:08:13.492 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:13 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:08:13.892 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:13.556+0000 7fea84f0c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T16:08:13.893 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:13.647+0000 7fea84f0c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T16:08:13.893 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:13.789+0000 7fea84f0c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T16:08:13.893 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:13.891+0000 7fea84f0c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T16:08:14.215 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:13 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:13.988+0000 7fea84f0c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T16:08:14.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:14 smithi086 ceph-mon[25584]: [04/Mar/2024:16:08:12] ENGINE Bus STARTING 2024-03-04T16:08:14.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:14 smithi086 ceph-mon[25584]: [04/Mar/2024:16:08:12] ENGINE Serving on https://172.21.15.83:7150 2024-03-04T16:08:14.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:14 smithi086 ceph-mon[25584]: [04/Mar/2024:16:08:12] ENGINE Serving on http://172.21.15.83:8765 2024-03-04T16:08:14.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:14 smithi086 ceph-mon[25584]: [04/Mar/2024:16:08:12] ENGINE Bus STARTED 2024-03-04T16:08:14.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:14 smithi086 ceph-mon[25584]: mgrmap e28: y(active, since 1.92403s) 2024-03-04T16:08:14.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:14 smithi086 ceph-mon[25584]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:08:14.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:14.260+0000 7fea84f0c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T16:08:14.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:14.374+0000 7fea84f0c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T16:08:14.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:14.461+0000 7fea84f0c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T16:08:14.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:14 smithi083 ceph-mon[24738]: [04/Mar/2024:16:08:12] ENGINE Bus STARTING 2024-03-04T16:08:14.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:14 smithi083 ceph-mon[24738]: [04/Mar/2024:16:08:12] ENGINE Serving on https://172.21.15.83:7150 2024-03-04T16:08:14.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:14 smithi083 ceph-mon[24738]: [04/Mar/2024:16:08:12] ENGINE Serving on http://172.21.15.83:8765 2024-03-04T16:08:14.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:14 smithi083 ceph-mon[24738]: [04/Mar/2024:16:08:12] ENGINE Bus STARTED 2024-03-04T16:08:14.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:14 smithi083 ceph-mon[24738]: mgrmap e28: y(active, since 1.92403s) 2024-03-04T16:08:14.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:14 smithi083 ceph-mon[24738]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:08:14.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:14 smithi083 ceph-mon[29452]: [04/Mar/2024:16:08:12] ENGINE Bus STARTING 2024-03-04T16:08:14.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:14 smithi083 ceph-mon[29452]: [04/Mar/2024:16:08:12] ENGINE Serving on https://172.21.15.83:7150 2024-03-04T16:08:14.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:14 smithi083 ceph-mon[29452]: [04/Mar/2024:16:08:12] ENGINE Serving on http://172.21.15.83:8765 2024-03-04T16:08:14.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:14 smithi083 ceph-mon[29452]: [04/Mar/2024:16:08:12] ENGINE Bus STARTED 2024-03-04T16:08:14.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:14 smithi083 ceph-mon[29452]: mgrmap e28: y(active, since 1.92403s) 2024-03-04T16:08:14.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:14 smithi083 ceph-mon[29452]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:08:14.984 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:14.557+0000 7fea84f0c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T16:08:14.984 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:14.762+0000 7fea84f0c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T16:08:15.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:15 smithi086 ceph-mon[25584]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:08:15.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:15 smithi083 ceph-mon[24738]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:08:15.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:15 smithi083 ceph-mon[29452]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:08:15.978 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:15 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:15.765+0000 7fea84f0c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T16:08:15.978 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:15 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:15.901+0000 7fea84f0c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T16:08:16.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:16 smithi086 ceph-mon[25584]: mgrmap e29: y(active, since 3s) 2024-03-04T16:08:16.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:16 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:16.259+0000 7fea84f0c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T16:08:16.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:16 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:16.352+0000 7fea84f0c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T16:08:16.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:16 smithi083 ceph-mon[29452]: mgrmap e29: y(active, since 3s) 2024-03-04T16:08:16.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:16 smithi083 ceph-mon[24738]: mgrmap e29: y(active, since 3s) 2024-03-04T16:08:16.984 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:16 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:16.573+0000 7fea84f0c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:08:16.984 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:16 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:16.763+0000 7fea84f0c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T16:08:17.350 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:17 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:17.082+0000 7fea84f0c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:08:17.604 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:17 smithi086 ceph-mon[25584]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:08:17.604 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:17 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:17.604 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:17 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:17.604 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:17 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:17.604 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:17 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:17.604 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:17 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:17.349+0000 7fea84f0c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T16:08:17.604 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:17 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:17.602+0000 7fea84f0c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T16:08:17.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:17 smithi083 ceph-mon[24738]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:08:17.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:17 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:17.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:17 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:17.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:17 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:17.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:17 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:17.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:17 smithi083 ceph-mon[29452]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:08:17.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:17 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:17.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:17 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:17.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:17 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:17.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:17 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:18.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:18 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:18.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:18 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:18.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:18 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:18.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:18 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:18.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:18 smithi086 ceph-mon[25584]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:08:18.735 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:18.445+0000 7fea84f0c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T16:08:18.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:08:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:08:18.853 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:18 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:18.853 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:18 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:18.853 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:18 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:18.853 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:18 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:18.853 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:18 smithi083 ceph-mon[24738]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:08:18.853 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:18 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:18.853 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:18 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:18.854 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:18 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:18.854 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:18 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:18.854 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:18 smithi083 ceph-mon[29452]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:08:19.348 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:19 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:19.019+0000 7fea84f0c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T16:08:19.348 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:19 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:19.122+0000 7fea84f0c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T16:08:19.348 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:19 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:19.344+0000 7fea84f0c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T16:08:19.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:19 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:08:19] ENGINE Bus STARTING 2024-03-04T16:08:19.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:19 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: CherryPy Checker: 2024-03-04T16:08:19.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:19 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: The Application mounted at '' has an empty config. 2024-03-04T16:08:19.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:19 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: 2024-03-04T16:08:19.735 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:19 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:08:19] ENGINE Serving on http://:::9283 2024-03-04T16:08:19.735 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:19 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: [04/Mar/2024:16:08:19] ENGINE Bus STARTED 2024-03-04T16:08:20.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:20 smithi083 ceph-mon[29452]: Standby manager daemon x started 2024-03-04T16:08:20.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:20 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/3985119790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T16:08:20.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:20 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/3985119790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:08:20.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:20 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/3985119790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T16:08:20.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:20 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/3985119790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:08:20.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:20 smithi083 ceph-mon[24738]: Standby manager daemon x started 2024-03-04T16:08:20.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:20 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/3985119790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T16:08:20.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:20 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/3985119790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:08:20.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:20 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/3985119790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T16:08:20.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:20 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/3985119790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:08:20.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:20 smithi086 ceph-mon[25584]: Standby manager daemon x started 2024-03-04T16:08:20.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:20 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/3985119790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T16:08:20.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:20 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/3985119790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:08:20.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:20 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/3985119790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T16:08:20.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:20 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/3985119790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:08:21.474 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:21 smithi083 ceph-mon[24738]: mgrmap e30: y(active, since 8s), standbys: x 2024-03-04T16:08:21.474 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:21 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T16:08:21.474 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:21 smithi083 ceph-mon[24738]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-04T16:08:21.474 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:21 smithi083 ceph-mon[29452]: mgrmap e30: y(active, since 8s), standbys: x 2024-03-04T16:08:21.474 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:21 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T16:08:21.474 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:21 smithi083 ceph-mon[29452]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-04T16:08:21.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:21 smithi086 ceph-mon[25584]: mgrmap e30: y(active, since 8s), standbys: x 2024-03-04T16:08:21.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:21 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T16:08:21.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:21 smithi086 ceph-mon[25584]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-04T16:08:22.347 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:22 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:22.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:22 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:22.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:22 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:22.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:22 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:08:22.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:22 smithi083 ceph-mon[29452]: Updating smithi083:/etc/ceph/ceph.conf 2024-03-04T16:08:22.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:22 smithi083 ceph-mon[29452]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-04T16:08:22.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:22 smithi083 ceph-mon[29452]: Updating smithi083:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:08:22.348 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:22 smithi083 ceph-mon[29452]: Updating smithi086:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:08:22.348 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:22 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:22.348 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:22 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:22.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:22 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:22.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:22 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:08:22.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:22 smithi083 ceph-mon[24738]: Updating smithi083:/etc/ceph/ceph.conf 2024-03-04T16:08:22.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:22 smithi083 ceph-mon[24738]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-04T16:08:22.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:22 smithi083 ceph-mon[24738]: Updating smithi083:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:08:22.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:22 smithi083 ceph-mon[24738]: Updating smithi086:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:08:22.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:22 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[46775]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:08:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-04T16:08:22.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:22 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:22.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:22 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:22.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:22 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:22.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:22 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:08:22.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:22 smithi086 ceph-mon[25584]: Updating smithi083:/etc/ceph/ceph.conf 2024-03-04T16:08:22.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:22 smithi086 ceph-mon[25584]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-04T16:08:22.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:22 smithi086 ceph-mon[25584]: Updating smithi083:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:08:22.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:22 smithi086 ceph-mon[25584]: Updating smithi086:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:08:23.329 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:23 smithi086 ceph-mon[25584]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:08:23.329 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:23 smithi086 ceph-mon[25584]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:08:23.329 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:23 smithi086 ceph-mon[25584]: Updating smithi086:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:08:23.330 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:23 smithi086 ceph-mon[25584]: Updating smithi083:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:08:23.330 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:23 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:23.330 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:23 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:23.330 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:23 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:23.330 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:23 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:23.330 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:23 smithi086 ceph-mon[25584]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-04T16:08:23.330 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:23 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:23.330 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:23 smithi086 ceph-mon[25584]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-04T16:08:23.330 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:23 smithi086 ceph-mon[25584]: Reconfiguring daemon prometheus.a on smithi086 2024-03-04T16:08:23.330 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:23 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/4044456800' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T16:08:23.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[24738]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:08:23.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[24738]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:08:23.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[24738]: Updating smithi086:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:08:23.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[24738]: Updating smithi083:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:08:23.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:23.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:23.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:23.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:23.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[24738]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-04T16:08:23.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:23.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[24738]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-04T16:08:23.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[24738]: Reconfiguring daemon prometheus.a on smithi086 2024-03-04T16:08:23.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/4044456800' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T16:08:23.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[29452]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:08:23.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[29452]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:08:23.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[29452]: Updating smithi086:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:08:23.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[29452]: Updating smithi083:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:08:23.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:23.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:23.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:23.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:23.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[29452]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-04T16:08:23.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:23.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[29452]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-04T16:08:23.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[29452]: Reconfiguring daemon prometheus.a on smithi086 2024-03-04T16:08:23.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:23 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/4044456800' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T16:08:24.480 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:24 smithi086 ceph-mon[25584]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-04T16:08:24.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:24 smithi083 ceph-mon[24738]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-04T16:08:24.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:24 smithi083 ceph-mon[29452]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-04T16:08:25.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:25 smithi086 ceph-mon[25584]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:08:25.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:25 smithi083 ceph-mon[24738]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:08:25.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:25 smithi083 ceph-mon[29452]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:08:25.878 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:25 smithi086 systemd[1]: Stopping Ceph prometheus.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:08:25.878 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:25 smithi086 bash[53695]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus.a" found: no such container 2024-03-04T16:08:26.147 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:25 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:08:25.878Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-04T16:08:26.147 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:25 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:08:25.878Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-04T16:08:26.147 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:25 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:08:25.878Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-04T16:08:26.147 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:25 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:08:25.878Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-04T16:08:26.147 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:25 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:08:25.878Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-04T16:08:26.147 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:25 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:08:25.878Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-04T16:08:26.147 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:25 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:08:25.878Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-04T16:08:26.147 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:25 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:08:25.878Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-04T16:08:26.147 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:25 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:08:25.878Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-04T16:08:26.147 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:25 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:08:25.879Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-04T16:08:26.147 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:25 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:08:25.879Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-04T16:08:26.148 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:25 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[50395]: ts=2024-03-04T16:08:25.879Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-04T16:08:26.148 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:25 smithi086 podman[53706]: 2024-03-04 16:08:25.888495382 +0000 UTC m=+0.129608597 container died f2ee5ff207554606a69c14ddd5e88f9b4abedfb85c1eaa9c0121c3ed2c3ce254 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:08:26.420 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 podman[53706]: 2024-03-04 16:08:26.227646972 +0000 UTC m=+0.468760194 container cleanup f2ee5ff207554606a69c14ddd5e88f9b4abedfb85c1eaa9c0121c3ed2c3ce254 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:08:26.420 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 bash[53706]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a 2024-03-04T16:08:26.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 podman[53720]: 2024-03-04 16:08:26.420452301 +0000 UTC m=+0.531113226 container remove f2ee5ff207554606a69c14ddd5e88f9b4abedfb85c1eaa9c0121c3ed2c3ce254 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:08:26.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 bash[53733]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus.a" found: no such container 2024-03-04T16:08:26.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-04T16:08:26.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 systemd[1]: Stopped Ceph prometheus.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:08:26.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 systemd[1]: Starting Ceph prometheus.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:08:26.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 podman[53802]: 2024-03-04 16:08:26.566512467 +0000 UTC m=+0.018725701 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-04T16:08:26.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 podman[53802]: 2024-03-04 16:08:26.663015471 +0000 UTC m=+0.115228704 container create b563a246eeac8f2acade5c23b28c5f50df43605df1eeb125e485ac1cd20e25e6 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:08:27.180 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 podman[53802]: 2024-03-04 16:08:26.932346921 +0000 UTC m=+0.384560161 container init b563a246eeac8f2acade5c23b28c5f50df43605df1eeb125e485ac1cd20e25e6 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:08:27.180 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 podman[53802]: 2024-03-04 16:08:26.935772497 +0000 UTC m=+0.387985739 container start b563a246eeac8f2acade5c23b28c5f50df43605df1eeb125e485ac1cd20e25e6 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:08:27.180 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:26.958Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-04T16:08:27.180 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:26.958Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-04T16:08:27.180 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:26.958Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-04T16:08:27.180 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:26.958Z caller=main.go:518 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi086 (none))" 2024-03-04T16:08:27.180 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:26.958Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-04T16:08:27.181 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:26.958Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-04T16:08:27.181 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:26.960Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-04T16:08:27.181 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:26.960Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-04T16:08:27.181 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:26.961Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-04T16:08:27.181 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:26.962Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-04T16:08:27.181 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:26.962Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.112µs 2024-03-04T16:08:27.181 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:26 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:26.962Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-04T16:08:27.181 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:27 smithi086 bash[53802]: b563a246eeac8f2acade5c23b28c5f50df43605df1eeb125e485ac1cd20e25e6 2024-03-04T16:08:27.181 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:27 smithi086 systemd[1]: Started Ceph prometheus.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:08:27.181 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:27.179Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-03-04T16:08:27.472 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:27.424Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-03-04T16:08:27.473 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:27 smithi086 ceph-mon[25584]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-04T16:08:27.473 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:27 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:27.473 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:27 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:27.473 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:27 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:08:27.473 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:27 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:27.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:27.472Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-03-04T16:08:27.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:27.472Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-03-04T16:08:27.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:27.472Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=41.218µs wal_replay_duration=510.10532ms total_replay_duration=510.165862ms 2024-03-04T16:08:27.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:27.476Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-04T16:08:27.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:27.476Z caller=main.go:947 level=info msg="TSDB started" 2024-03-04T16:08:27.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:27.476Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-04T16:08:27.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:27.497Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.835788ms db_storage=860ns remote_storage=1.308µs web_handler=390ns query_engine=720ns scrape=231.922µs scrape_sd=93.843µs notify=13.19µs notify_sd=19.106µs rules=20.210306ms 2024-03-04T16:08:27.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:27.497Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-04T16:08:27.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:27 smithi083 ceph-mon[29452]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-04T16:08:27.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:27 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:27.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:27 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:27.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:27 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:08:27.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:27 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:27.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:27 smithi083 ceph-mon[24738]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-04T16:08:27.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:27 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:27.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:27 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:27.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:27 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:08:27.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:27 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:28.630 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:28 smithi086 ceph-mon[25584]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:08:28.630 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:28 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:28.630 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:28 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:28.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:28 smithi083 ceph-mon[24738]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:08:28.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:28 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:28.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:28 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:28.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:28 smithi083 ceph-mon[29452]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:08:28.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:28 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:28.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:28 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:29.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:29 smithi086 ceph-mon[25584]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-04T16:08:29.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:29 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:08:29.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:29 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:08:29.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:29 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:08:29.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:29 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:29.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:29 smithi083 ceph-mon[29452]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-04T16:08:29.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:29 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:08:29.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:29 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:08:29.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:29 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:08:29.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:29 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:29.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:29 smithi083 ceph-mon[24738]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-04T16:08:29.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:29 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:08:29.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:29 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:08:29.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:29 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:08:29.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:29 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:30.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:30 smithi086 ceph-mon[25584]: Upgrade: Updating mgr.x 2024-03-04T16:08:30.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:30 smithi086 ceph-mon[25584]: Deploying daemon mgr.x on smithi086 2024-03-04T16:08:30.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:30 smithi083 ceph-mon[29452]: Upgrade: Updating mgr.x 2024-03-04T16:08:30.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:30 smithi083 ceph-mon[29452]: Deploying daemon mgr.x on smithi086 2024-03-04T16:08:30.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:30 smithi083 ceph-mon[24738]: Upgrade: Updating mgr.x 2024-03-04T16:08:30.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:30 smithi083 ceph-mon[24738]: Deploying daemon mgr.x on smithi086 2024-03-04T16:08:30.779 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:30 smithi086 systemd[1]: Stopping Ceph mgr.x for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:08:31.084 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:30 smithi086 podman[54156]: 2024-03-04 16:08:30.779813304 +0000 UTC m=+0.149411275 container died a73348a7968ee1fa7b096bc4a81a6e44f3ccd788938c8b8de2aee68b540cc45e (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-04T16:08:31.433 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:31 smithi086 ceph-mon[25584]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:08:31.434 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:31 smithi086 podman[54156]: 2024-03-04 16:08:31.174247822 +0000 UTC m=+0.543845801 container cleanup a73348a7968ee1fa7b096bc4a81a6e44f3ccd788938c8b8de2aee68b540cc45e (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-04T16:08:31.434 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:31 smithi086 bash[54156]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x 2024-03-04T16:08:31.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:31 smithi086 podman[54170]: 2024-03-04 16:08:31.433806718 +0000 UTC m=+0.656747679 container remove a73348a7968ee1fa7b096bc4a81a6e44f3ccd788938c8b8de2aee68b540cc45e (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-04T16:08:31.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:31 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-04T16:08:31.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:31 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-04T16:08:31.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:31 smithi086 systemd[1]: Stopped Ceph mgr.x for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:08:31.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:31 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mgr.x.service: Consumed 26.632s CPU time. 2024-03-04T16:08:31.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:31 smithi083 ceph-mon[29452]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:08:31.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:31 smithi083 ceph-mon[24738]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:08:32.132 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:31 smithi086 systemd[1]: Starting Ceph mgr.x for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:08:32.132 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:31 smithi086 podman[54298]: 2024-03-04 16:08:31.901413749 +0000 UTC m=+0.017356055 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:08:32.132 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:32 smithi086 podman[54298]: 2024-03-04 16:08:32.010082227 +0000 UTC m=+0.126024520 container create 3c33100cb8a899af8230b0790d80e5c17d66903c91eb394ac0ce6be286ef587d (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:08:32.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:32 smithi086 podman[54298]: 2024-03-04 16:08:32.228973039 +0000 UTC m=+0.344915336 container init 3c33100cb8a899af8230b0790d80e5c17d66903c91eb394ac0ce6be286ef587d (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux ) 2024-03-04T16:08:32.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:32 smithi086 podman[54298]: 2024-03-04 16:08:32.231655577 +0000 UTC m=+0.347597873 container start 3c33100cb8a899af8230b0790d80e5c17d66903c91eb394ac0ce6be286ef587d (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, ceph=True) 2024-03-04T16:08:32.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:32 smithi086 ceph-mgr[54317]: -- 172.21.15.86:0/4034759824 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d832db85a0 con 0x55d832d8d000 2024-03-04T16:08:32.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:32 smithi086 bash[54298]: 3c33100cb8a899af8230b0790d80e5c17d66903c91eb394ac0ce6be286ef587d 2024-03-04T16:08:32.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:32 smithi086 systemd[1]: Started Ceph mgr.x for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:08:33.144 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:32 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:32.786+0000 7f0a96e80200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T16:08:33.144 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:32 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:32.888+0000 7f0a96e80200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T16:08:33.667 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:33 smithi086 ceph-mon[25584]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:08:33.667 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:33 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:33.667 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:33 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:33.667 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:33.562+0000 7f0a96e80200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T16:08:33.667 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:33.665+0000 7f0a96e80200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T16:08:33.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:33 smithi083 ceph-mon[29452]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:08:33.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:33 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:33.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:33 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:33.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:33 smithi083 ceph-mon[24738]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:08:33.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:33 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:33.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:33 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:33.984 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:33.811+0000 7f0a96e80200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T16:08:33.984 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:33.910+0000 7f0a96e80200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T16:08:34.271 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:33.999+0000 7f0a96e80200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T16:08:34.271 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:34.270+0000 7f0a96e80200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T16:08:34.596 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:34 smithi086 ceph-mon[25584]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:34.596 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:34.392+0000 7f0a96e80200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T16:08:34.597 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:34.490+0000 7f0a96e80200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T16:08:34.597 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:34.595+0000 7f0a96e80200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T16:08:34.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:34 smithi083 ceph-mon[24738]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:34.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:34 smithi083 ceph-mon[29452]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:34.882 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:08:34.895 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:34 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:34.776+0000 7f0a96e80200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T16:08:35.524 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:08:35.524 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (81s) 18s ago 6m 14.8M - 0.23.0 ba2b418f427c 3c6dc3779a0e 2024-03-04T16:08:35.524 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (43s) 18s ago 6m 44.3M - 8.3.5 dad864ee21e9 fb754d58c53f 2024-03-04T16:08:35.524 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 unknown 18s ago 5m - - 2024-03-04T16:08:35.524 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443,9283,8765 starting - - - - 2024-03-04T16:08:35.524 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:8443,9283,8765 running (38s) 18s ago 12m 475M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 164dd93609ee 2024-03-04T16:08:35.524 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (12m) 18s ago 12m 54.3M 2048M 17.2.0 e1d6a67b021e d2ba878ed364 2024-03-04T16:08:35.524 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (11m) 18s ago 11m 43.8M 2048M 17.2.0 e1d6a67b021e d1b972ebdd2a 2024-03-04T16:08:35.524 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (11m) 18s ago 11m 43.9M 2048M 17.2.0 e1d6a67b021e e74b0ab73a80 2024-03-04T16:08:35.524 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (93s) 18s ago 7m 10.9M - 1.3.1 1dbe0e931976 9bf54b460640 2024-03-04T16:08:35.525 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (62s) 18s ago 7m 9.89M - 1.3.1 1dbe0e931976 b72dd80b2588 2024-03-04T16:08:35.525 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (10m) 18s ago 10m 52.0M 2736M 17.2.0 e1d6a67b021e 56679a9f787b 2024-03-04T16:08:35.525 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (10m) 18s ago 10m 55.0M 2736M 17.2.0 e1d6a67b021e c1070c1a315a 2024-03-04T16:08:35.525 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (9m) 18s ago 9m 48.1M 2736M 17.2.0 e1d6a67b021e 2b913c19f581 2024-03-04T16:08:35.525 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (9m) 18s ago 9m 50.3M 2736M 17.2.0 e1d6a67b021e 9f9a4e2ec5da 2024-03-04T16:08:35.525 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (9m) 18s ago 9m 52.9M 2992M 17.2.0 e1d6a67b021e b3a97f83f1c7 2024-03-04T16:08:35.525 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (8m) 18s ago 8m 50.7M 2992M 17.2.0 e1d6a67b021e b82ca42b7353 2024-03-04T16:08:35.525 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (8m) 18s ago 8m 49.0M 2992M 17.2.0 e1d6a67b021e 18cedf14321d 2024-03-04T16:08:35.525 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (7m) 18s ago 7m 53.3M 2992M 17.2.0 e1d6a67b021e 390e1798c738 2024-03-04T16:08:35.525 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 starting - - - - 2024-03-04T16:08:35.525 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (5m) 18s ago 5m 72.5M - 17.2.0 e1d6a67b021e 65f14042f354 2024-03-04T16:08:35.525 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (5m) 18s ago 5m 73.7M - 17.2.0 e1d6a67b021e 78175cc6dc5d 2024-03-04T16:08:35.525 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (5m) 18s ago 5m 70.8M - 17.2.0 e1d6a67b021e 01dcf858bd62 2024-03-04T16:08:35.525 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (5m) 18s ago 5m 75.6M - 17.2.0 e1d6a67b021e 8f70a5ce9f0d 2024-03-04T16:08:35.689 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:35 smithi086 ceph-mon[25584]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-04T16:08:35.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:35 smithi083 ceph-mon[24738]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-04T16:08:35.729 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:35 smithi083 ceph-mon[29452]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-04T16:08:35.907 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:08:35.907 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:08:35.908 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-04T16:08:35.908 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:08:35.908 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:08:35.908 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:08:35.908 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:08:35.908 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:08:35.908 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-04T16:08:35.908 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:08:35.908 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": {}, 2024-03-04T16:08:35.908 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:08:35.908 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-04T16:08:35.908 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:08:35.908 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:08:35.908 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-03-04T16:08:35.909 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:08:35.909 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:08:35.909 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:08:35.984 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:35 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:35.780+0000 7f0a96e80200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T16:08:35.984 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:35 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:35.915+0000 7f0a96e80200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T16:08:35.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:35 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:08:35] "GET /metrics HTTP/1.1" 200 38231 "" "Prometheus/2.33.4" 2024-03-04T16:08:36.232 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:08:36.232 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:08:36.232 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:08:36.232 INFO:teuthology.orchestra.run.smithi083.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:08:36.233 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [], 2024-03-04T16:08:36.233 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "", 2024-03-04T16:08:36.233 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "Currently upgrading mgr daemons", 2024-03-04T16:08:36.233 INFO:teuthology.orchestra.run.smithi083.stdout: "is_paused": false 2024-03-04T16:08:36.233 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:08:36.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:36 smithi086 ceph-mon[25584]: from='client.25327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:36.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:36 smithi086 ceph-mon[25584]: from='client.15429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:36.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:36 smithi086 ceph-mon[25584]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:36.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:36 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/1833367536' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:36.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:36 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:36.273+0000 7f0a96e80200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T16:08:36.485 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:36 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:36.367+0000 7f0a96e80200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T16:08:36.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:36 smithi083 ceph-mon[29452]: from='client.25327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:36.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:36 smithi083 ceph-mon[29452]: from='client.15429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:36.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:36 smithi083 ceph-mon[29452]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:36.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:36 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/1833367536' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:36.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:36 smithi083 ceph-mon[24738]: from='client.25327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:36.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:36 smithi083 ceph-mon[24738]: from='client.15429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:36.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:36 smithi083 ceph-mon[24738]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:36.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:36 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/1833367536' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:36.621 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-04T16:08:36.621 INFO:teuthology.orchestra.run.smithi083.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-04T16:08:36.621 INFO:teuthology.orchestra.run.smithi083.stdout: daemon iscsi.foo.smithi083.fnypow on smithi083 is in unknown state 2024-03-04T16:08:36.984 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:36 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:36.568+0000 7f0a96e80200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:08:36.984 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:36 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:36.770+0000 7f0a96e80200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T16:08:37.350 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:37 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:37.074+0000 7f0a96e80200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:08:37.350 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:37 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:37.348+0000 7f0a96e80200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T16:08:37.603 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:37 smithi086 ceph-mon[25584]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:37.604 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:37 smithi086 ceph-mon[25584]: from='client.25351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:37.604 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:37 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/2890215970' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:08:37.604 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:37 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:37.604 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:37 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:37.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:37 smithi083 ceph-mon[29452]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:37.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:37 smithi083 ceph-mon[29452]: from='client.25351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:37.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:37 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/2890215970' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:08:37.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:37 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:37.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:37 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:37.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:37 smithi083 ceph-mon[24738]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:37.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:37 smithi083 ceph-mon[24738]: from='client.25351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:08:37.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:37 smithi083 ceph-mon[24738]: from='client.? 172.21.15.83:0/2890215970' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:08:37.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:37 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:37.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:37 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:37.984 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:37 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:37.602+0000 7f0a96e80200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T16:08:38.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:38 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:38.472+0000 7f0a96e80200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T16:08:38.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:38 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:08:39.168 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:38 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:39.168 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:38 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:39.168 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:38 smithi086 ceph-mon[25584]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:39.169 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:39 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:39.066+0000 7f0a96e80200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T16:08:39.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:38 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:39.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:38 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:39.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:38 smithi083 ceph-mon[29452]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:39.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:38 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:39.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:38 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:39.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:38 smithi083 ceph-mon[24738]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:39.430 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:39 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:39.167+0000 7f0a96e80200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T16:08:39.430 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:39 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:39.397+0000 7f0a96e80200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T16:08:39.430 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:39 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: [04/Mar/2024:16:08:39] ENGINE Bus STARTING 2024-03-04T16:08:39.688 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:39 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: CherryPy Checker: 2024-03-04T16:08:39.688 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:39 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: The Application mounted at '' has an empty config. 2024-03-04T16:08:39.688 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:39 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:39.688 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:39 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: [04/Mar/2024:16:08:39] ENGINE Serving on http://:::9283 2024-03-04T16:08:39.688 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:39 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: [04/Mar/2024:16:08:39] ENGINE Bus STARTED 2024-03-04T16:08:39.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:39 smithi086 ceph-mon[25584]: Standby manager daemon x restarted 2024-03-04T16:08:39.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:39 smithi086 ceph-mon[25584]: Standby manager daemon x started 2024-03-04T16:08:39.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:39 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/978798118' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T16:08:39.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:39 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/978798118' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:08:39.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:39 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/978798118' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T16:08:39.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:39 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/978798118' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:08:40.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:39 smithi083 ceph-mon[24738]: Standby manager daemon x restarted 2024-03-04T16:08:40.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:39 smithi083 ceph-mon[24738]: Standby manager daemon x started 2024-03-04T16:08:40.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:39 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/978798118' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T16:08:40.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:39 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/978798118' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:08:40.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:39 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/978798118' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T16:08:40.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:39 smithi083 ceph-mon[24738]: from='mgr.? 172.21.15.86:0/978798118' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:08:40.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:39 smithi083 ceph-mon[29452]: Standby manager daemon x restarted 2024-03-04T16:08:40.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:39 smithi083 ceph-mon[29452]: Standby manager daemon x started 2024-03-04T16:08:40.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:39 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/978798118' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T16:08:40.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:39 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/978798118' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:08:40.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:39 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/978798118' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T16:08:40.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:39 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/978798118' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:08:41.140 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:40 smithi086 ceph-mon[25584]: mgrmap e31: y(active, since 28s), standbys: x 2024-03-04T16:08:41.141 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:40 smithi086 ceph-mon[25584]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:08:41.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:40 smithi083 ceph-mon[24738]: mgrmap e31: y(active, since 28s), standbys: x 2024-03-04T16:08:41.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:40 smithi083 ceph-mon[24738]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:08:41.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:40 smithi083 ceph-mon[29452]: mgrmap e31: y(active, since 28s), standbys: x 2024-03-04T16:08:41.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:40 smithi083 ceph-mon[29452]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:08:42.610 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:42.610 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:42.610 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:42.610 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:08:42.610 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[24738]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 903 B/s rd, 0 op/s 2024-03-04T16:08:42.610 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[24738]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:08:42.610 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:42.610 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:42.610 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:42.610 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:42.611 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-04T16:08:42.611 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-04T16:08:42.611 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-04T16:08:42.611 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-04T16:08:42.611 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-04T16:08:42.611 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-04T16:08:42.611 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-04T16:08:42.611 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-04T16:08:42.611 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:42.611 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:42.611 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:42.612 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:42.612 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:08:42.612 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[29452]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 903 B/s rd, 0 op/s 2024-03-04T16:08:42.612 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[29452]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:08:42.612 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:42.612 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:42.612 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:42.612 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:42.612 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-04T16:08:42.612 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-04T16:08:42.612 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-04T16:08:42.612 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-04T16:08:42.612 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-04T16:08:42.613 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-04T16:08:42.613 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-04T16:08:42.613 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-04T16:08:42.613 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:42 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:42.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:42 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:42.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:42 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:42.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:42 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:42.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:42 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:08:42.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:42 smithi086 ceph-mon[25584]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 903 B/s rd, 0 op/s 2024-03-04T16:08:42.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:42 smithi086 ceph-mon[25584]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:08:42.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:42 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:42.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:42 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:42.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:42 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:08:42.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:42 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:42.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:42 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-04T16:08:42.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:42 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-04T16:08:42.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:42 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-04T16:08:42.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:42 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-04T16:08:42.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:42 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-04T16:08:42.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:42 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-04T16:08:42.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:42 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-04T16:08:42.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:42 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-04T16:08:42.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:42 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:43.865 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:43 smithi083 ceph-mon[24738]: Upgrade: Setting container_image for all mgr 2024-03-04T16:08:43.865 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:43 smithi083 ceph-mon[24738]: Upgrade: It appears safe to stop mon.a 2024-03-04T16:08:43.865 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:43 smithi083 ceph-mon[24738]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:43.865 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:43 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:08:43.865 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:43 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:08:43.866 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:43 smithi083 ceph-mon[24738]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:43.866 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:43 smithi083 ceph-mon[29452]: Upgrade: Setting container_image for all mgr 2024-03-04T16:08:43.866 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:43 smithi083 ceph-mon[29452]: Upgrade: It appears safe to stop mon.a 2024-03-04T16:08:43.866 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:43 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:43.866 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:43 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:08:43.866 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:43 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:08:43.866 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:43 smithi083 ceph-mon[29452]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:43.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:43 smithi086 ceph-mon[25584]: Upgrade: Setting container_image for all mgr 2024-03-04T16:08:43.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:43 smithi086 ceph-mon[25584]: Upgrade: It appears safe to stop mon.a 2024-03-04T16:08:43.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:43 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='mgr.y' 2024-03-04T16:08:43.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:43 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:08:43.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:43 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:08:43.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:43 smithi086 ceph-mon[25584]: from='mgr.25231 172.21.15.83:0/1676648712' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:44.762 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:44 smithi083 ceph-mon[29452]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:44.762 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:44 smithi083 ceph-mon[29452]: Upgrade: Updating mon.a 2024-03-04T16:08:44.762 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:44 smithi083 ceph-mon[29452]: Deploying daemon mon.a on smithi083 2024-03-04T16:08:44.762 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:44 smithi083 ceph-mon[29452]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:08:44.763 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:44 smithi083 ceph-mon[24738]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:44.763 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:44 smithi083 ceph-mon[24738]: Upgrade: Updating mon.a 2024-03-04T16:08:44.763 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:44 smithi083 ceph-mon[24738]: Deploying daemon mon.a on smithi083 2024-03-04T16:08:44.763 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:44 smithi083 ceph-mon[24738]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:08:44.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:44 smithi086 ceph-mon[25584]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:44.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:44 smithi086 ceph-mon[25584]: Upgrade: Updating mon.a 2024-03-04T16:08:44.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:44 smithi086 ceph-mon[25584]: Deploying daemon mon.a on smithi083 2024-03-04T16:08:44.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:44 smithi086 ceph-mon[25584]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:08:45.309 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:45 smithi083 systemd[1]: Stopping Ceph mon.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:08:45.634 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a[24734]: 2024-03-04T16:08:45.308+0000 7f209d7dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:08:45.635 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a[24734]: 2024-03-04T16:08:45.308+0000 7f209d7dc700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-04T16:08:45.635 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:45 smithi083 podman[64679]: 2024-03-04 16:08:45.380242473 +0000 UTC m=+0.182414726 container died d2ba878ed364f67a2e89a9d4c420c99fef93063d13a2339c7871b8c0076cf018 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, maintainer=Guillaume Abrioux , name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, GIT_CLEAN=True, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, build-date=2022-05-03T08:36:31.336870, release=754) 2024-03-04T16:08:45.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:08:45] "GET /metrics HTTP/1.1" 200 38230 "" "Prometheus/2.33.4" 2024-03-04T16:08:45.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:45 smithi083 podman[64679]: 2024-03-04 16:08:45.725684192 +0000 UTC m=+0.527856493 container cleanup d2ba878ed364f67a2e89a9d4c420c99fef93063d13a2339c7871b8c0076cf018 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a, release=754, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, ceph=True, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, distribution-scope=public, version=8, RELEASE=HEAD, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, name=centos-stream, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-04T16:08:45.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:45 smithi083 bash[64679]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a 2024-03-04T16:08:46.362 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 podman[64693]: 2024-03-04 16:08:46.00192552 +0000 UTC m=+0.622993061 container remove d2ba878ed364f67a2e89a9d4c420c99fef93063d13a2339c7871b8c0076cf018 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, release=754, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, io.openshift.tags=base centos centos-stream, distribution-scope=public, ceph=True) 2024-03-04T16:08:46.362 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-04T16:08:46.362 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 systemd[1]: Stopped Ceph mon.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:08:46.362 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mon.a.service: Consumed 13.648s CPU time. 2024-03-04T16:08:46.703 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 systemd[1]: Starting Ceph mon.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:08:46.703 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 podman[64825]: 2024-03-04 16:08:46.477677631 +0000 UTC m=+0.017495433 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:08:46.703 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 podman[64825]: 2024-03-04 16:08:46.594997099 +0000 UTC m=+0.134814893 container create c173a244219755ddf31bc89aef1bb32a59ef13edc120c8eeaae95d7f32389eb2 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS) 2024-03-04T16:08:46.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 podman[64825]: 2024-03-04 16:08:46.799510147 +0000 UTC m=+0.339327959 container init c173a244219755ddf31bc89aef1bb32a59ef13edc120c8eeaae95d7f32389eb2 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-04T16:08:46.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 podman[64825]: 2024-03-04 16:08:46.802845985 +0000 UTC m=+0.342663789 container start c173a244219755ddf31bc89aef1bb32a59ef13edc120c8eeaae95d7f32389eb2 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-04T16:08:46.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: set uid:gid to 167:167 (ceph:ceph) 2024-03-04T16:08:46.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev), process ceph-mon, pid 2 2024-03-04T16:08:46.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: pidfile_write: ignore empty --pid-file 2024-03-04T16:08:46.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: load: jerasure load: lrc 2024-03-04T16:08:46.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: RocksDB version: 7.9.2 2024-03-04T16:08:46.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Git sha 0 2024-03-04T16:08:46.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Compile date 2024-03-04 11:37:16 2024-03-04T16:08:46.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: DB SUMMARY 2024-03-04T16:08:46.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: DB Session ID: P7HG55NAJWU5Q3BPP17W 2024-03-04T16:08:46.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: CURRENT file: CURRENT 2024-03-04T16:08:46.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: IDENTITY file: IDENTITY 2024-03-04T16:08:46.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: MANIFEST file: MANIFEST-000015 size: 972 Bytes 2024-03-04T16:08:46.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000030.sst 2024-03-04T16:08:46.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000028.log size: 3604129 ; 2024-03-04T16:08:46.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.error_if_exists: 0 2024-03-04T16:08:46.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.create_if_missing: 0 2024-03-04T16:08:46.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.paranoid_checks: 1 2024-03-04T16:08:46.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-04T16:08:46.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-04T16:08:46.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-04T16:08:46.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.env: 0x560de4cfeb20 2024-03-04T16:08:46.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.fs: PosixFileSystem 2024-03-04T16:08:46.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.info_log: 0x560de631d8e0 2024-03-04T16:08:46.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_file_opening_threads: 16 2024-03-04T16:08:46.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.statistics: (nil) 2024-03-04T16:08:46.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.use_fsync: 0 2024-03-04T16:08:46.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_log_file_size: 0 2024-03-04T16:08:46.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-04T16:08:46.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-04T16:08:46.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.keep_log_file_num: 1000 2024-03-04T16:08:46.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.recycle_log_file_num: 0 2024-03-04T16:08:46.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.allow_fallocate: 1 2024-03-04T16:08:46.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.allow_mmap_reads: 0 2024-03-04T16:08:46.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.allow_mmap_writes: 0 2024-03-04T16:08:46.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.use_direct_reads: 0 2024-03-04T16:08:46.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-04T16:08:46.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.create_missing_column_families: 0 2024-03-04T16:08:46.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.db_log_dir: 2024-03-04T16:08:46.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.wal_dir: 2024-03-04T16:08:46.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-04T16:08:46.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-04T16:08:46.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-04T16:08:46.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-04T16:08:46.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-04T16:08:46.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-04T16:08:46.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.advise_random_on_open: 1 2024-03-04T16:08:46.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.db_write_buffer_size: 0 2024-03-04T16:08:46.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.write_buffer_manager: 0x560de62f1400 2024-03-04T16:08:46.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-04T16:08:46.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-04T16:08:46.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-04T16:08:46.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.rate_limiter: (nil) 2024-03-04T16:08:46.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-04T16:08:46.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.wal_recovery_mode: 2 2024-03-04T16:08:46.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.enable_thread_tracking: 0 2024-03-04T16:08:46.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.enable_pipelined_write: 0 2024-03-04T16:08:46.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.unordered_write: 0 2024-03-04T16:08:46.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-04T16:08:46.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-04T16:08:46.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-04T16:08:46.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-04T16:08:46.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.row_cache: None 2024-03-04T16:08:46.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.wal_filter: None 2024-03-04T16:08:46.994 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-04T16:08:46.995 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.allow_ingest_behind: 0 2024-03-04T16:08:46.995 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.two_write_queues: 0 2024-03-04T16:08:46.995 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.manual_wal_flush: 0 2024-03-04T16:08:46.995 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.wal_compression: 0 2024-03-04T16:08:46.995 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.atomic_flush: 0 2024-03-04T16:08:46.995 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-04T16:08:46.995 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-04T16:08:46.995 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-04T16:08:46.995 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.log_readahead_size: 0 2024-03-04T16:08:46.995 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-04T16:08:46.995 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.best_efforts_recovery: 0 2024-03-04T16:08:46.995 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-04T16:08:46.996 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-04T16:08:46.996 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.allow_data_in_errors: 0 2024-03-04T16:08:46.996 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.db_host_id: __hostname__ 2024-03-04T16:08:46.996 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-04T16:08:46.996 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_background_jobs: 2 2024-03-04T16:08:46.996 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_background_compactions: -1 2024-03-04T16:08:46.996 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_subcompactions: 1 2024-03-04T16:08:46.996 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-04T16:08:46.996 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-04T16:08:46.996 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-04T16:08:46.996 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_total_wal_size: 0 2024-03-04T16:08:46.997 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-04T16:08:46.997 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-04T16:08:46.997 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-04T16:08:46.997 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-04T16:08:46.997 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_open_files: -1 2024-03-04T16:08:46.997 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.bytes_per_sync: 0 2024-03-04T16:08:46.997 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-04T16:08:46.997 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-04T16:08:46.997 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compaction_readahead_size: 0 2024-03-04T16:08:46.997 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_background_flushes: -1 2024-03-04T16:08:46.997 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Compression algorithms supported: 2024-03-04T16:08:46.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-04T16:08:46.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: kZSTD supported: 0 2024-03-04T16:08:46.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: kXpressCompression supported: 0 2024-03-04T16:08:46.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: kLZ4HCCompression supported: 1 2024-03-04T16:08:46.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: kZlibCompression supported: 1 2024-03-04T16:08:46.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: kSnappyCompression supported: 1 2024-03-04T16:08:46.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: kLZ4Compression supported: 1 2024-03-04T16:08:46.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: kBZip2Compression supported: 0 2024-03-04T16:08:46.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-04T16:08:46.999 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-04T16:08:46.999 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-03-04T16:08:46.999 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-04T16:08:46.999 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-04T16:08:46.999 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.merge_operator: 2024-03-04T16:08:46.999 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compaction_filter: None 2024-03-04T16:08:46.999 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compaction_filter_factory: None 2024-03-04T16:08:46.999 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.sst_partitioner_factory: None 2024-03-04T16:08:46.999 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-04T16:08:46.999 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-04T16:08:46.999 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560de631d9e0) 2024-03-04T16:08:46.999 INFO:journalctl@ceph.mon.a.smithi083.stdout: cache_index_and_filter_blocks: 1 2024-03-04T16:08:46.999 INFO:journalctl@ceph.mon.a.smithi083.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-04T16:08:46.999 INFO:journalctl@ceph.mon.a.smithi083.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-04T16:08:47.000 INFO:journalctl@ceph.mon.a.smithi083.stdout: pin_top_level_index_and_filter: 1 2024-03-04T16:08:47.000 INFO:journalctl@ceph.mon.a.smithi083.stdout: index_type: 0 2024-03-04T16:08:47.000 INFO:journalctl@ceph.mon.a.smithi083.stdout: data_block_index_type: 0 2024-03-04T16:08:47.000 INFO:journalctl@ceph.mon.a.smithi083.stdout: index_shortening: 1 2024-03-04T16:08:47.000 INFO:journalctl@ceph.mon.a.smithi083.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-04T16:08:47.000 INFO:journalctl@ceph.mon.a.smithi083.stdout: checksum: 4 2024-03-04T16:08:47.000 INFO:journalctl@ceph.mon.a.smithi083.stdout: no_block_cache: 0 2024-03-04T16:08:47.000 INFO:journalctl@ceph.mon.a.smithi083.stdout: block_cache: 0x560de6323090 2024-03-04T16:08:47.000 INFO:journalctl@ceph.mon.a.smithi083.stdout: block_cache_name: BinnedLRUCache 2024-03-04T16:08:47.000 INFO:journalctl@ceph.mon.a.smithi083.stdout: block_cache_options: 2024-03-04T16:08:47.000 INFO:journalctl@ceph.mon.a.smithi083.stdout: capacity : 536870912 2024-03-04T16:08:47.000 INFO:journalctl@ceph.mon.a.smithi083.stdout: num_shard_bits : 4 2024-03-04T16:08:47.000 INFO:journalctl@ceph.mon.a.smithi083.stdout: strict_capacity_limit : 0 2024-03-04T16:08:47.000 INFO:journalctl@ceph.mon.a.smithi083.stdout: high_pri_pool_ratio: 0.000 2024-03-04T16:08:47.001 INFO:journalctl@ceph.mon.a.smithi083.stdout: block_cache_compressed: (nil) 2024-03-04T16:08:47.001 INFO:journalctl@ceph.mon.a.smithi083.stdout: persistent_cache: (nil) 2024-03-04T16:08:47.001 INFO:journalctl@ceph.mon.a.smithi083.stdout: block_size: 4096 2024-03-04T16:08:47.001 INFO:journalctl@ceph.mon.a.smithi083.stdout: block_size_deviation: 10 2024-03-04T16:08:47.001 INFO:journalctl@ceph.mon.a.smithi083.stdout: block_restart_interval: 16 2024-03-04T16:08:47.001 INFO:journalctl@ceph.mon.a.smithi083.stdout: index_block_restart_interval: 1 2024-03-04T16:08:47.001 INFO:journalctl@ceph.mon.a.smithi083.stdout: metadata_block_size: 4096 2024-03-04T16:08:47.001 INFO:journalctl@ceph.mon.a.smithi083.stdout: partition_filters: 0 2024-03-04T16:08:47.001 INFO:journalctl@ceph.mon.a.smithi083.stdout: use_delta_encoding: 1 2024-03-04T16:08:47.001 INFO:journalctl@ceph.mon.a.smithi083.stdout: filter_policy: bloomfilter 2024-03-04T16:08:47.001 INFO:journalctl@ceph.mon.a.smithi083.stdout: whole_key_filtering: 1 2024-03-04T16:08:47.001 INFO:journalctl@ceph.mon.a.smithi083.stdout: verify_compression: 0 2024-03-04T16:08:47.001 INFO:journalctl@ceph.mon.a.smithi083.stdout: read_amp_bytes_per_bit: 0 2024-03-04T16:08:47.001 INFO:journalctl@ceph.mon.a.smithi083.stdout: format_version: 5 2024-03-04T16:08:47.001 INFO:journalctl@ceph.mon.a.smithi083.stdout: enable_index_compression: 1 2024-03-04T16:08:47.002 INFO:journalctl@ceph.mon.a.smithi083.stdout: block_align: 0 2024-03-04T16:08:47.002 INFO:journalctl@ceph.mon.a.smithi083.stdout: max_auto_readahead_size: 262144 2024-03-04T16:08:47.002 INFO:journalctl@ceph.mon.a.smithi083.stdout: prepopulate_block_cache: 0 2024-03-04T16:08:47.002 INFO:journalctl@ceph.mon.a.smithi083.stdout: initial_auto_readahead_size: 8192 2024-03-04T16:08:47.002 INFO:journalctl@ceph.mon.a.smithi083.stdout: num_file_reads_for_auto_readahead: 2 2024-03-04T16:08:47.002 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.write_buffer_size: 33554432 2024-03-04T16:08:47.002 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_write_buffer_number: 2 2024-03-04T16:08:47.002 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compression: NoCompression 2024-03-04T16:08:47.002 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.bottommost_compression: Disabled 2024-03-04T16:08:47.002 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.prefix_extractor: nullptr 2024-03-04T16:08:47.002 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-04T16:08:47.002 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.num_levels: 7 2024-03-04T16:08:47.002 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-04T16:08:47.003 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-04T16:08:47.003 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-04T16:08:47.003 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-04T16:08:47.003 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-04T16:08:47.003 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-04T16:08:47.003 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-04T16:08:47.003 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-04T16:08:47.003 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-04T16:08:47.003 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-04T16:08:47.003 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-04T16:08:47.003 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-04T16:08:47.003 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-04T16:08:47.003 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compression_opts.level: 32767 2024-03-04T16:08:47.003 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compression_opts.strategy: 0 2024-03-04T16:08:47.004 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-04T16:08:47.004 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-04T16:08:47.004 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-04T16:08:47.004 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-04T16:08:47.004 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compression_opts.enabled: false 2024-03-04T16:08:47.004 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-04T16:08:47.004 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-04T16:08:47.004 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-04T16:08:47.004 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-04T16:08:47.004 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.target_file_size_base: 67108864 2024-03-04T16:08:47.004 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-04T16:08:47.004 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-04T16:08:47.004 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-04T16:08:47.005 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-04T16:08:47.005 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-04T16:08:47.005 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-04T16:08:47.005 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-04T16:08:47.005 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-04T16:08:47.005 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-04T16:08:47.005 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-04T16:08:47.005 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-04T16:08:47.005 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-04T16:08:47.005 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-04T16:08:47.005 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-04T16:08:47.005 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.arena_block_size: 1048576 2024-03-04T16:08:47.005 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-04T16:08:47.005 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-04T16:08:47.005 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.disable_auto_compactions: 0 2024-03-04T16:08:47.006 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-04T16:08:47.006 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-04T16:08:47.006 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-04T16:08:47.006 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-04T16:08:47.006 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-04T16:08:47.006 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-04T16:08:47.006 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-04T16:08:47.006 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-04T16:08:47.006 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-04T16:08:47.006 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-04T16:08:47.006 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.table_properties_collectors: 2024-03-04T16:08:47.006 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.inplace_update_support: 0 2024-03-04T16:08:47.006 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-04T16:08:47.006 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-04T16:08:47.006 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-04T16:08:47.007 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-04T16:08:47.007 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.bloom_locality: 0 2024-03-04T16:08:47.007 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.max_successive_merges: 0 2024-03-04T16:08:47.007 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-04T16:08:47.007 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.paranoid_file_checks: 0 2024-03-04T16:08:47.007 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.force_consistency_checks: 1 2024-03-04T16:08:47.007 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.report_bg_io_stats: 0 2024-03-04T16:08:47.007 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.ttl: 2592000 2024-03-04T16:08:47.007 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-04T16:08:47.007 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-04T16:08:47.007 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-04T16:08:47.007 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.enable_blob_files: false 2024-03-04T16:08:47.007 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.min_blob_size: 0 2024-03-04T16:08:47.007 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.blob_file_size: 268435456 2024-03-04T16:08:47.007 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-04T16:08:47.008 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-04T16:08:47.008 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-04T16:08:47.008 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-04T16:08:47.008 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-04T16:08:47.008 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.blob_file_starting_level: 0 2024-03-04T16:08:47.008 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-04T16:08:47.008 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-03-04T16:08:47.008 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-04T16:08:47.008 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 12110, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-04T16:08:47.008 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-03-04T16:08:47.008 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1a4d993a-4266-494c-a91d-b730cf20813b 2024-03-04T16:08:47.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709568526832807, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-03-04T16:08:47.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-03-04T16:08:47.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709568526842287, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 3319473, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12111, "largest_seqno": 13773, "table_properties": {"data_size": 3311740, "index_size": 4857, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1861, "raw_key_size": 16951, "raw_average_key_size": 23, "raw_value_size": 3296189, "raw_average_value_size": 4584, "num_data_blocks": 220, "num_entries": 719, "num_filter_entries": 719, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709568526, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1a4d993a-4266-494c-a91d-b730cf20813b", "db_session_id": "P7HG55NAJWU5Q3BPP17W", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-03-04T16:08:47.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709568526842402, "job": 1, "event": "recovery_finished"} 2024-03-04T16:08:47.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-03-04T16:08:47.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-04T16:08:47.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-04T16:08:47.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560de6416000 2024-03-04T16:08:47.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: DB pointer 0x560de6400000 2024-03-04T16:08:47.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-04T16:08:47.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-04T16:08:47.009 INFO:journalctl@ceph.mon.a.smithi083.stdout: ** DB Stats ** 2024-03-04T16:08:47.010 INFO:journalctl@ceph.mon.a.smithi083.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T16:08:47.010 INFO:journalctl@ceph.mon.a.smithi083.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-04T16:08:47.010 INFO:journalctl@ceph.mon.a.smithi083.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T16:08:47.010 INFO:journalctl@ceph.mon.a.smithi083.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T16:08:47.010 INFO:journalctl@ceph.mon.a.smithi083.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-04T16:08:47.010 INFO:journalctl@ceph.mon.a.smithi083.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T16:08:47.010 INFO:journalctl@ceph.mon.a.smithi083.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T16:08:47.010 INFO:journalctl@ceph.mon.a.smithi083.stdout: 2024-03-04T16:08:47.010 INFO:journalctl@ceph.mon.a.smithi083.stdout: ** Compaction Stats [default] ** 2024-03-04T16:08:47.010 INFO:journalctl@ceph.mon.a.smithi083.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T16:08:47.010 INFO:journalctl@ceph.mon.a.smithi083.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-04T16:08:47.010 INFO:journalctl@ceph.mon.a.smithi083.stdout: L0 1/0 3.17 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 708.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-04T16:08:47.010 INFO:journalctl@ceph.mon.a.smithi083.stdout: L6 1/0 7.58 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-04T16:08:47.011 INFO:journalctl@ceph.mon.a.smithi083.stdout: Sum 2/0 10.75 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 708.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-04T16:08:47.011 INFO:journalctl@ceph.mon.a.smithi083.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 708.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-04T16:08:47.011 INFO:journalctl@ceph.mon.a.smithi083.stdout: 2024-03-04T16:08:47.011 INFO:journalctl@ceph.mon.a.smithi083.stdout: ** Compaction Stats [default] ** 2024-03-04T16:08:47.011 INFO:journalctl@ceph.mon.a.smithi083.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T16:08:47.011 INFO:journalctl@ceph.mon.a.smithi083.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T16:08:47.011 INFO:journalctl@ceph.mon.a.smithi083.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 708.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-04T16:08:47.011 INFO:journalctl@ceph.mon.a.smithi083.stdout: 2024-03-04T16:08:47.011 INFO:journalctl@ceph.mon.a.smithi083.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-04T16:08:47.011 INFO:journalctl@ceph.mon.a.smithi083.stdout: 2024-03-04T16:08:47.011 INFO:journalctl@ceph.mon.a.smithi083.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T16:08:47.011 INFO:journalctl@ceph.mon.a.smithi083.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-03-04T16:08:47.011 INFO:journalctl@ceph.mon.a.smithi083.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T16:08:47.011 INFO:journalctl@ceph.mon.a.smithi083.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T16:08:47.012 INFO:journalctl@ceph.mon.a.smithi083.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T16:08:47.012 INFO:journalctl@ceph.mon.a.smithi083.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T16:08:47.012 INFO:journalctl@ceph.mon.a.smithi083.stdout: Cumulative compaction: 0.00 GB write, 263.76 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T16:08:47.012 INFO:journalctl@ceph.mon.a.smithi083.stdout: Interval compaction: 0.00 GB write, 263.76 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T16:08:47.012 INFO:journalctl@ceph.mon.a.smithi083.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T16:08:47.012 INFO:journalctl@ceph.mon.a.smithi083.stdout: Block cache BinnedLRUCache@0x560de6323090#2 capacity: 512.00 MB usage: 7.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 2e-05 secs_since: 0 2024-03-04T16:08:47.012 INFO:journalctl@ceph.mon.a.smithi083.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,5.08 KB,0.000968575%) Misc(1,0.00 KB,0%) 2024-03-04T16:08:47.012 INFO:journalctl@ceph.mon.a.smithi083.stdout: 2024-03-04T16:08:47.012 INFO:journalctl@ceph.mon.a.smithi083.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T16:08:47.012 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: starting mon.a rank 0 at public addrs [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] at bind addrs [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T16:08:47.012 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: mon.a@-1(???) e3 preinit fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T16:08:47.012 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: mon.a@-1(???).mds e1 new map 2024-03-04T16:08:47.012 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: mon.a@-1(???).mds e1 print_map 2024-03-04T16:08:47.012 INFO:journalctl@ceph.mon.a.smithi083.stdout: e1 2024-03-04T16:08:47.013 INFO:journalctl@ceph.mon.a.smithi083.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:08:47.013 INFO:journalctl@ceph.mon.a.smithi083.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:08:47.013 INFO:journalctl@ceph.mon.a.smithi083.stdout: legacy client fscid: -1 2024-03-04T16:08:47.013 INFO:journalctl@ceph.mon.a.smithi083.stdout: 2024-03-04T16:08:47.013 INFO:journalctl@ceph.mon.a.smithi083.stdout: No filesystems configured 2024-03-04T16:08:47.013 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: mon.a@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-04T16:08:47.013 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-04T16:08:47.013 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-04T16:08:47.013 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-04T16:08:47.013 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 ceph-mon[64844]: mon.a@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-03-04T16:08:47.013 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 bash[64825]: c173a244219755ddf31bc89aef1bb32a59ef13edc120c8eeaae95d7f32389eb2 2024-03-04T16:08:47.013 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:46 smithi083 systemd[1]: Started Ceph mon.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:08:48.984 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:08:48 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:08:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:08:50.234 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:49 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: ignoring --setuser ceph since I am not root 2024-03-04T16:08:50.234 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:49 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: ignoring --setgroup ceph since I am not root 2024-03-04T16:08:50.234 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:49 smithi086 ceph-mgr[54317]: -- 172.21.15.86:0/2021235495 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bf8f46c5a0 con 0x55bf8f440c00 2024-03-04T16:08:50.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:49 smithi086 ceph-mon[25584]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:08:50.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:49 smithi086 ceph-mon[25584]: mon.a calling monitor election 2024-03-04T16:08:50.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:49 smithi086 ceph-mon[25584]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:08:50.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:49 smithi086 ceph-mon[25584]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:08:50.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:49 smithi086 ceph-mon[25584]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T16:08:50.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:49 smithi086 ceph-mon[25584]: monmap e3: 3 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],b=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],c=[v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T16:08:50.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:49 smithi086 ceph-mon[25584]: fsmap 2024-03-04T16:08:50.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:49 smithi086 ceph-mon[25584]: osdmap e105: 8 total, 8 up, 8 in 2024-03-04T16:08:50.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:49 smithi086 ceph-mon[25584]: mgrmap e31: y(active, since 38s), standbys: x 2024-03-04T16:08:50.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:49 smithi086 ceph-mon[25584]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-04T16:08:50.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:49 smithi086 ceph-mon[25584]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-04T16:08:50.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:49 smithi086 ceph-mon[25584]: daemon iscsi.foo.smithi083.fnypow on smithi083 is in unknown state 2024-03-04T16:08:50.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:49 smithi086 ceph-mon[25584]: from='mgr.25231 ' entity='' 2024-03-04T16:08:50.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:49 smithi086 ceph-mon[25584]: mgrmap e32: y(active, since 38s), standbys: x 2024-03-04T16:08:50.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ignoring --setuser ceph since I am not root 2024-03-04T16:08:50.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ignoring --setgroup ceph since I am not root 2024-03-04T16:08:50.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mgr[60488]: -- 172.21.15.83:0/1850239993 <== mon.1 v2:172.21.15.83:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56474b8f45a0 con 0x56474b8c8800 2024-03-04T16:08:50.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[29452]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:08:50.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[29452]: mon.a calling monitor election 2024-03-04T16:08:50.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[29452]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:08:50.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[29452]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:08:50.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[29452]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T16:08:50.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[29452]: monmap e3: 3 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],b=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],c=[v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T16:08:50.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[29452]: fsmap 2024-03-04T16:08:50.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[29452]: osdmap e105: 8 total, 8 up, 8 in 2024-03-04T16:08:50.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[29452]: mgrmap e31: y(active, since 38s), standbys: x 2024-03-04T16:08:50.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[29452]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-04T16:08:50.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[29452]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-04T16:08:50.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[29452]: daemon iscsi.foo.smithi083.fnypow on smithi083 is in unknown state 2024-03-04T16:08:50.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[29452]: from='mgr.25231 ' entity='' 2024-03-04T16:08:50.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[29452]: mgrmap e32: y(active, since 38s), standbys: x 2024-03-04T16:08:50.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[64844]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:08:50.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[64844]: mon.a calling monitor election 2024-03-04T16:08:50.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[64844]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:08:50.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[64844]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:08:50.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[64844]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T16:08:50.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[64844]: monmap e3: 3 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],b=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],c=[v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T16:08:50.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[64844]: fsmap 2024-03-04T16:08:50.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[64844]: osdmap e105: 8 total, 8 up, 8 in 2024-03-04T16:08:50.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[64844]: mgrmap e31: y(active, since 38s), standbys: x 2024-03-04T16:08:50.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[64844]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-04T16:08:50.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[64844]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-04T16:08:50.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[64844]: daemon iscsi.foo.smithi083.fnypow on smithi083 is in unknown state 2024-03-04T16:08:50.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[64844]: from='mgr.25231 ' entity='' 2024-03-04T16:08:50.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:49 smithi083 ceph-mon[64844]: mgrmap e32: y(active, since 38s), standbys: x 2024-03-04T16:08:50.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:50 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:50.471+0000 7fad1215f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T16:08:50.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:50 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:50.565+0000 7fad1215f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T16:08:50.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:50 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:50.458+0000 7fe8acbce200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T16:08:50.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:50 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:50.553+0000 7fe8acbce200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T16:08:51.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:51.207+0000 7fad1215f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T16:08:51.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:51.300+0000 7fad1215f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T16:08:51.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:51.442+0000 7fad1215f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T16:08:51.488 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:51 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:51.197+0000 7fe8acbce200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T16:08:51.488 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:51 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:51.289+0000 7fe8acbce200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T16:08:51.488 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:51 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:51.430+0000 7fe8acbce200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T16:08:51.875 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:51 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:51.526+0000 7fe8acbce200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T16:08:51.875 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:51 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:51.609+0000 7fe8acbce200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T16:08:51.886 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:51.537+0000 7fad1215f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T16:08:51.886 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:51.620+0000 7fad1215f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T16:08:51.886 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:51.886+0000 7fad1215f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T16:08:52.166 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:51 smithi083 ceph-mon[29452]: mgrmap e33: y(active, since 39s), standbys: x 2024-03-04T16:08:52.166 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:51 smithi083 ceph-mon[64844]: mgrmap e33: y(active, since 39s), standbys: x 2024-03-04T16:08:52.167 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:51 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:51.874+0000 7fe8acbce200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T16:08:52.167 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:51 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:51.985+0000 7fe8acbce200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T16:08:52.167 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:52 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:52.070+0000 7fe8acbce200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T16:08:52.167 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:52 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:52.165+0000 7fe8acbce200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T16:08:52.179 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:51 smithi086 ceph-mon[25584]: mgrmap e33: y(active, since 39s), standbys: x 2024-03-04T16:08:52.179 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:51.996+0000 7fad1215f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T16:08:52.179 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:52.084+0000 7fad1215f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T16:08:52.179 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:52.178+0000 7fad1215f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T16:08:52.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:52 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:52.355+0000 7fad1215f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T16:08:52.487 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:52 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:52.342+0000 7fe8acbce200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T16:08:53.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:53 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:53.313+0000 7fad1215f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T16:08:53.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:53 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:53.442+0000 7fad1215f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T16:08:53.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:53 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:53.299+0000 7fe8acbce200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T16:08:53.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:53 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:53.429+0000 7fe8acbce200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T16:08:54.057 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:53 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:53.765+0000 7fe8acbce200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T16:08:54.058 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:53 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:53.858+0000 7fe8acbce200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T16:08:54.068 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:53 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:53.776+0000 7fad1215f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T16:08:54.068 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:53 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:53.868+0000 7fad1215f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T16:08:54.068 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:54 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:54.067+0000 7fad1215f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:08:54.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:54 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:54.255+0000 7fad1215f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T16:08:54.487 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:54 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:54.057+0000 7fe8acbce200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:08:54.488 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:54 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:54.244+0000 7fe8acbce200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T16:08:54.795 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:54 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:54.532+0000 7fe8acbce200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:08:54.805 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:54 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:54.544+0000 7fad1215f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:08:54.805 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:54 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:54.804+0000 7fad1215f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T16:08:55.234 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:55 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:55.044+0000 7fad1215f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T16:08:55.237 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:54 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:54.794+0000 7fe8acbce200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T16:08:55.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:55.037+0000 7fe8acbce200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T16:08:56.135 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:55 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:55.860+0000 7fad1215f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T16:08:56.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:55.841+0000 7fe8acbce200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T16:08:56.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:56.456+0000 7fad1215f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T16:08:56.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:56.563+0000 7fad1215f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T16:08:56.737 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:56.416+0000 7fe8acbce200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T16:08:56.738 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:56.517+0000 7fe8acbce200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T16:08:57.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: Active manager daemon y restarted 2024-03-04T16:08:57.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: Activating manager daemon y 2024-03-04T16:08:57.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: osdmap e106: 8 total, 8 up, 8 in 2024-03-04T16:08:57.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: mgrmap e34: y(active, starting, since 0.00508104s), standbys: x 2024-03-04T16:08:57.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:08:57.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:08:57.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:08:57.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T16:08:57.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T16:08:57.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:08:57.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:08:57.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:08:57.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:08:57.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:08:57.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:08:57.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:08:57.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:08:57.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:08:57.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:08:57.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:08:57.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-mon[25584]: Manager daemon y is now available 2024-03-04T16:08:57.236 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:56.798+0000 7fad1215f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T16:08:57.236 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: [04/Mar/2024:16:08:56] ENGINE Bus STARTING 2024-03-04T16:08:57.236 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: CherryPy Checker: 2024-03-04T16:08:57.236 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: The Application mounted at '' has an empty config. 2024-03-04T16:08:57.236 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: 2024-03-04T16:08:57.236 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: [04/Mar/2024:16:08:56] ENGINE Serving on http://:::9283 2024-03-04T16:08:57.236 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:08:56 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x[54313]: [04/Mar/2024:16:08:56] ENGINE Bus STARTED 2024-03-04T16:08:57.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:56.742+0000 7fe8acbce200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T16:08:57.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: [04/Mar/2024:16:08:56] ENGINE Bus STARTING 2024-03-04T16:08:57.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: CherryPy Checker: 2024-03-04T16:08:57.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: The Application mounted at '' has an empty config. 2024-03-04T16:08:57.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:57.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: [04/Mar/2024:16:08:56] ENGINE Serving on http://:::9283 2024-03-04T16:08:57.239 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: [04/Mar/2024:16:08:56] ENGINE Bus STARTED 2024-03-04T16:08:57.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: Active manager daemon y restarted 2024-03-04T16:08:57.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: Activating manager daemon y 2024-03-04T16:08:57.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: osdmap e106: 8 total, 8 up, 8 in 2024-03-04T16:08:57.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: mgrmap e34: y(active, starting, since 0.00508104s), standbys: x 2024-03-04T16:08:57.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:08:57.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:08:57.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:08:57.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T16:08:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T16:08:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:08:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:08:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:08:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:08:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:08:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:08:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:08:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:08:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:08:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:08:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:08:57.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[29452]: Manager daemon y is now available 2024-03-04T16:08:57.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: Active manager daemon y restarted 2024-03-04T16:08:57.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: Activating manager daemon y 2024-03-04T16:08:57.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: osdmap e106: 8 total, 8 up, 8 in 2024-03-04T16:08:57.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: mgrmap e34: y(active, starting, since 0.00508104s), standbys: x 2024-03-04T16:08:57.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:08:57.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:08:57.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:08:57.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T16:08:57.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T16:08:57.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:08:57.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:08:57.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:08:57.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:08:57.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:08:57.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:08:57.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:08:57.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:08:57.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:08:57.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:08:57.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:08:57.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:56 smithi083 ceph-mon[64844]: Manager daemon y is now available 2024-03-04T16:08:58.098 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:08:57.775+0000 7fe862dc1700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-04T16:08:58.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[64844]: Standby manager daemon x restarted 2024-03-04T16:08:58.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[64844]: Standby manager daemon x started 2024-03-04T16:08:58.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[64844]: from='mgr.? 172.21.15.86:0/2453017353' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T16:08:58.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[64844]: from='mgr.? 172.21.15.86:0/2453017353' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:08:58.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[64844]: from='mgr.? 172.21.15.86:0/2453017353' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T16:08:58.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[64844]: from='mgr.? 172.21.15.86:0/2453017353' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:08:58.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:58.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:58.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:08:58.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:08:58.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[64844]: mgrmap e35: y(active, since 1.01193s), standbys: x 2024-03-04T16:08:58.099 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[29452]: Standby manager daemon x restarted 2024-03-04T16:08:58.099 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[29452]: Standby manager daemon x started 2024-03-04T16:08:58.100 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/2453017353' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T16:08:58.100 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/2453017353' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:08:58.100 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/2453017353' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T16:08:58.100 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[29452]: from='mgr.? 172.21.15.86:0/2453017353' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:08:58.100 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:58.100 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:58.100 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:08:58.100 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:08:58.100 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:57 smithi083 ceph-mon[29452]: mgrmap e35: y(active, since 1.01193s), standbys: x 2024-03-04T16:08:58.117 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:57 smithi086 ceph-mon[25584]: Standby manager daemon x restarted 2024-03-04T16:08:58.117 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:57 smithi086 ceph-mon[25584]: Standby manager daemon x started 2024-03-04T16:08:58.117 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:57 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/2453017353' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T16:08:58.117 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:57 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/2453017353' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:08:58.117 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:57 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/2453017353' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T16:08:58.117 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:57 smithi086 ceph-mon[25584]: from='mgr.? 172.21.15.86:0/2453017353' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:08:58.118 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:57 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:58.118 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:57 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:58.118 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:57 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:08:58.118 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:57 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:08:58.118 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:57 smithi086 ceph-mon[25584]: mgrmap e35: y(active, since 1.01193s), standbys: x 2024-03-04T16:08:58.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:58 smithi086 ceph-mon[25584]: [04/Mar/2024:16:08:57] ENGINE Bus STARTING 2024-03-04T16:08:58.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:58 smithi086 ceph-mon[25584]: [04/Mar/2024:16:08:57] ENGINE Serving on http://172.21.15.83:8765 2024-03-04T16:08:58.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:58 smithi086 ceph-mon[25584]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-04T16:08:58.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:58 smithi086 ceph-mon[25584]: Cluster is now healthy 2024-03-04T16:08:59.072 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:58 smithi083 ceph-mon[64844]: [04/Mar/2024:16:08:57] ENGINE Bus STARTING 2024-03-04T16:08:59.072 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:58 smithi083 ceph-mon[64844]: [04/Mar/2024:16:08:57] ENGINE Serving on http://172.21.15.83:8765 2024-03-04T16:08:59.072 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:58 smithi083 ceph-mon[64844]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-04T16:08:59.072 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:58 smithi083 ceph-mon[64844]: Cluster is now healthy 2024-03-04T16:08:59.072 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:58 smithi083 ceph-mon[29452]: [04/Mar/2024:16:08:57] ENGINE Bus STARTING 2024-03-04T16:08:59.072 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:58 smithi083 ceph-mon[29452]: [04/Mar/2024:16:08:57] ENGINE Serving on http://172.21.15.83:8765 2024-03-04T16:08:59.073 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:58 smithi083 ceph-mon[29452]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-04T16:08:59.073 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:58 smithi083 ceph-mon[29452]: Cluster is now healthy 2024-03-04T16:08:59.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:59 smithi083 ceph-mon[29452]: [04/Mar/2024:16:08:58] ENGINE Serving on https://172.21.15.83:7150 2024-03-04T16:08:59.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:59 smithi083 ceph-mon[29452]: [04/Mar/2024:16:08:58] ENGINE Bus STARTED 2024-03-04T16:08:59.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:59 smithi083 ceph-mon[29452]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:08:59.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:08:59 smithi083 ceph-mon[29452]: mgrmap e36: y(active, since 2s), standbys: x 2024-03-04T16:08:59.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:59 smithi083 ceph-mon[64844]: [04/Mar/2024:16:08:58] ENGINE Serving on https://172.21.15.83:7150 2024-03-04T16:08:59.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:59 smithi083 ceph-mon[64844]: [04/Mar/2024:16:08:58] ENGINE Bus STARTED 2024-03-04T16:08:59.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:59 smithi083 ceph-mon[64844]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:08:59.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:08:59 smithi083 ceph-mon[64844]: mgrmap e36: y(active, since 2s), standbys: x 2024-03-04T16:09:00.170 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:59 smithi086 ceph-mon[25584]: [04/Mar/2024:16:08:58] ENGINE Serving on https://172.21.15.83:7150 2024-03-04T16:09:00.170 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:59 smithi086 ceph-mon[25584]: [04/Mar/2024:16:08:58] ENGINE Bus STARTED 2024-03-04T16:09:00.170 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:59 smithi086 ceph-mon[25584]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:09:00.170 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:08:59 smithi086 ceph-mon[25584]: mgrmap e36: y(active, since 2s), standbys: x 2024-03-04T16:09:02.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:01 smithi086 ceph-mon[25584]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:09:02.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:01 smithi086 ceph-mon[25584]: mgrmap e37: y(active, since 4s), standbys: x 2024-03-04T16:09:02.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:01 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:02.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:01 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:02.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:01 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:02.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:01 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:02.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:01 smithi083 ceph-mon[29452]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:09:02.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:01 smithi083 ceph-mon[29452]: mgrmap e37: y(active, since 4s), standbys: x 2024-03-04T16:09:02.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:01 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:02.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:01 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:02.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:01 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:02.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:01 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:02.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:01 smithi083 ceph-mon[64844]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:09:02.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:01 smithi083 ceph-mon[64844]: mgrmap e37: y(active, since 4s), standbys: x 2024-03-04T16:09:02.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:01 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:02.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:01 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:02.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:01 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:02.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:01 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:03.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:03 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:03.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:03 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:03.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:03 smithi086 ceph-mon[25584]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:09:03.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:03 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:03.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:03 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:03.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:03 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:03.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:03 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:03.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:03 smithi083 ceph-mon[64844]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:09:03.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:03 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:03.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:03 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:03.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:03 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:03.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:03 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:03.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:03 smithi083 ceph-mon[29452]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:09:03.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:03 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:03.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:03 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:05.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:05 smithi086 ceph-mon[25584]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-04T16:09:05.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:05 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:05.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:05 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:05.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:05 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:05.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:05 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:09:05.987 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:09:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:09:05] "GET /metrics HTTP/1.1" 200 35440 "" "Prometheus/2.33.4" 2024-03-04T16:09:05.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:05 smithi083 ceph-mon[29452]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-04T16:09:05.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:05 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:05.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:05 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:05.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:05 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:05.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:05 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:09:05.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:05 smithi083 ceph-mon[64844]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-04T16:09:05.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:05 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:05.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:05 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:05.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:05 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:05.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:05 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:09:06.990 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:09:07.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:06 smithi086 ceph-mon[25584]: Updating smithi083:/etc/ceph/ceph.conf 2024-03-04T16:09:07.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:06 smithi086 ceph-mon[25584]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-04T16:09:07.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:06 smithi086 ceph-mon[25584]: Updating smithi086:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:09:07.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:06 smithi086 ceph-mon[25584]: Updating smithi083:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:09:07.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:06 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:07.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:06 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:07.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:06 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:07.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:06 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:07.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:06 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:07.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:06 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:09:07.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:06 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:07.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:06 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-04T16:09:07.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:06 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-04T16:09:07.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[29452]: Updating smithi083:/etc/ceph/ceph.conf 2024-03-04T16:09:07.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[29452]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-04T16:09:07.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[29452]: Updating smithi086:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:09:07.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[29452]: Updating smithi083:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:09:07.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:07.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:07.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:07.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:07.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:07.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:09:07.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:07.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-04T16:09:07.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-04T16:09:07.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[64844]: Updating smithi083:/etc/ceph/ceph.conf 2024-03-04T16:09:07.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[64844]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-04T16:09:07.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[64844]: Updating smithi086:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:09:07.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[64844]: Updating smithi083:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:09:07.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:07.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:07.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:07.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:07.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:07.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:09:07.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:07.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-04T16:09:07.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:06 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-04T16:09:07.668 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:09:07.668 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (113s) 5s ago 7m 15.7M - 0.23.0 ba2b418f427c 3c6dc3779a0e 2024-03-04T16:09:07.668 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (76s) 6s ago 6m 44.5M - 8.3.5 dad864ee21e9 fb754d58c53f 2024-03-04T16:09:07.668 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 running (45s) 5s ago 5m 48.3M - 3.5 e1d6a67b021e d58e71ccf9db 2024-03-04T16:09:07.668 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443,9283,8765 running (35s) 6s ago 11m 447M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c33100cb8a8 2024-03-04T16:09:07.668 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:8443,9283,8765 running (70s) 5s ago 13m 472M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 164dd93609ee 2024-03-04T16:09:07.669 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (21s) 5s ago 13m 28.8M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c173a2442197 2024-03-04T16:09:07.669 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (11m) 6s ago 11m 50.2M 2048M 17.2.0 e1d6a67b021e d1b972ebdd2a 2024-03-04T16:09:07.669 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (11m) 5s ago 11m 48.9M 2048M 17.2.0 e1d6a67b021e e74b0ab73a80 2024-03-04T16:09:07.669 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (2m) 5s ago 7m 18.2M - 1.3.1 1dbe0e931976 9bf54b460640 2024-03-04T16:09:07.669 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (94s) 6s ago 7m 15.8M - 1.3.1 1dbe0e931976 b72dd80b2588 2024-03-04T16:09:07.669 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (11m) 5s ago 11m 52.7M 2736M 17.2.0 e1d6a67b021e 56679a9f787b 2024-03-04T16:09:07.669 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (10m) 5s ago 10m 55.7M 2736M 17.2.0 e1d6a67b021e c1070c1a315a 2024-03-04T16:09:07.669 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (10m) 5s ago 10m 48.8M 2736M 17.2.0 e1d6a67b021e 2b913c19f581 2024-03-04T16:09:07.669 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (9m) 5s ago 9m 50.8M 2736M 17.2.0 e1d6a67b021e 9f9a4e2ec5da 2024-03-04T16:09:07.669 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (9m) 6s ago 9m 53.3M 2992M 17.2.0 e1d6a67b021e b3a97f83f1c7 2024-03-04T16:09:07.669 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (9m) 6s ago 9m 50.2M 2992M 17.2.0 e1d6a67b021e b82ca42b7353 2024-03-04T16:09:07.669 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (8m) 6s ago 8m 49.7M 2992M 17.2.0 e1d6a67b021e 18cedf14321d 2024-03-04T16:09:07.669 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (8m) 6s ago 8m 54.1M 2992M 17.2.0 e1d6a67b021e 390e1798c738 2024-03-04T16:09:07.669 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 running (41s) 6s ago 7m 53.7M - 2.33.4 514e6a882f6e b563a246eeac 2024-03-04T16:09:07.670 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (6m) 5s ago 6m 73.3M - 17.2.0 e1d6a67b021e 65f14042f354 2024-03-04T16:09:07.670 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (6m) 6s ago 6m 74.6M - 17.2.0 e1d6a67b021e 78175cc6dc5d 2024-03-04T16:09:07.670 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (6m) 5s ago 6m 71.5M - 17.2.0 e1d6a67b021e 01dcf858bd62 2024-03-04T16:09:07.670 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (5m) 6s ago 5m 76.3M - 17.2.0 e1d6a67b021e 8f70a5ce9f0d 2024-03-04T16:09:07.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:07 smithi083 ceph-mon[29452]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:09:07.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:07 smithi083 ceph-mon[29452]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:09:07.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:07 smithi083 ceph-mon[29452]: Updating smithi086:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:09:07.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:07 smithi083 ceph-mon[29452]: Updating smithi083:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:09:07.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:07 smithi083 ceph-mon[29452]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-04T16:09:07.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:07 smithi083 ceph-mon[29452]: Upgrade: It appears safe to stop mon.c 2024-03-04T16:09:07.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:07 smithi083 ceph-mon[64844]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:09:07.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:07 smithi083 ceph-mon[64844]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:09:07.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:07 smithi083 ceph-mon[64844]: Updating smithi086:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:09:07.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:07 smithi083 ceph-mon[64844]: Updating smithi083:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:09:07.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:07 smithi083 ceph-mon[64844]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-04T16:09:07.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:07 smithi083 ceph-mon[64844]: Upgrade: It appears safe to stop mon.c 2024-03-04T16:09:08.081 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:09:08.081 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:09:08.081 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-04T16:09:08.081 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 1 2024-03-04T16:09:08.082 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:09:08.082 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:09:08.082 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:09:08.082 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:09:08.082 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:09:08.082 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-04T16:09:08.082 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:09:08.082 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": {}, 2024-03-04T16:09:08.082 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:09:08.082 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-04T16:09:08.082 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:09:08.082 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:09:08.082 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-03-04T16:09:08.082 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 3 2024-03-04T16:09:08.083 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:09:08.083 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:09:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:07 smithi086 ceph-mon[25584]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:09:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:07 smithi086 ceph-mon[25584]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:09:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:07 smithi086 ceph-mon[25584]: Updating smithi086:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:09:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:07 smithi086 ceph-mon[25584]: Updating smithi083:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:09:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:07 smithi086 ceph-mon[25584]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-04T16:09:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:07 smithi086 ceph-mon[25584]: Upgrade: It appears safe to stop mon.c 2024-03-04T16:09:08.426 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:09:08.426 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:09:08.426 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:09:08.427 INFO:teuthology.orchestra.run.smithi083.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:09:08.427 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [ 2024-03-04T16:09:08.427 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr" 2024-03-04T16:09:08.427 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-04T16:09:08.427 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "3/23 daemons upgraded", 2024-03-04T16:09:08.427 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "Currently upgrading mon daemons", 2024-03-04T16:09:08.427 INFO:teuthology.orchestra.run.smithi083.stdout: "is_paused": false 2024-03-04T16:09:08.427 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:09:08.839 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_OK 2024-03-04T16:09:08.984 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:09:08 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:09:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:09:08.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:08 smithi086 ceph-mon[25584]: from='client.25399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:08.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:08 smithi086 ceph-mon[25584]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:08.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:08 smithi086 ceph-mon[25584]: from='client.44148 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:08.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:08 smithi086 ceph-mon[25584]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:08.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:08 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/3122950173' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:08.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:08 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:08.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:08 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:09:08.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:08 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:09:08.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:08 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:09.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:08 smithi083 ceph-mon[29452]: from='client.25399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:09.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:08 smithi083 ceph-mon[29452]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:09.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:08 smithi083 ceph-mon[29452]: from='client.44148 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:09.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:08 smithi083 ceph-mon[29452]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:09.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:08 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/3122950173' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:09.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:08 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:09.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:08 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:09:09.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:08 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:09:09.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:08 smithi083 ceph-mon[29452]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:09.089 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:08 smithi083 ceph-mon[64844]: from='client.25399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:09.089 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:08 smithi083 ceph-mon[64844]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:09.089 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:08 smithi083 ceph-mon[64844]: from='client.44148 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:09.089 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:08 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:09.089 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:08 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/3122950173' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:09.089 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:08 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:09.089 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:08 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:09:09.089 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:08 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:09:09.089 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:08 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:10.105 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:09 smithi083 ceph-mon[64844]: Upgrade: Updating mon.c 2024-03-04T16:09:10.105 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:09 smithi083 ceph-mon[64844]: Deploying daemon mon.c on smithi083 2024-03-04T16:09:10.105 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:09 smithi083 ceph-mon[64844]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:10.105 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:09 smithi083 ceph-mon[64844]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-04T16:09:10.105 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:09 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/4113794158' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:09:10.105 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:09 smithi083 ceph-mon[29452]: Upgrade: Updating mon.c 2024-03-04T16:09:10.105 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:09 smithi083 ceph-mon[29452]: Deploying daemon mon.c on smithi083 2024-03-04T16:09:10.105 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:09 smithi083 ceph-mon[29452]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:10.106 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:09 smithi083 ceph-mon[29452]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-04T16:09:10.106 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:09 smithi083 ceph-mon[29452]: from='client.? 172.21.15.83:0/4113794158' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:09:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:09 smithi086 ceph-mon[25584]: Upgrade: Updating mon.c 2024-03-04T16:09:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:09 smithi086 ceph-mon[25584]: Deploying daemon mon.c on smithi083 2024-03-04T16:09:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:09 smithi086 ceph-mon[25584]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:09 smithi086 ceph-mon[25584]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-04T16:09:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:09 smithi086 ceph-mon[25584]: from='client.? 172.21.15.83:0/4113794158' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:09:10.398 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:10 smithi083 systemd[1]: Stopping Ceph mon.c for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:09:10.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:10 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-c[29448]: 2024-03-04T16:09:10.397+0000 7fab7799b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:09:10.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:10 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-c[29448]: 2024-03-04T16:09:10.397+0000 7fab7799b700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-04T16:09:10.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:10 smithi083 podman[67215]: 2024-03-04 16:09:10.551149479 +0000 UTC m=+0.301982148 container died e74b0ab73a802d289c44e66156695991f2d41618264fc9971a39b124060b467e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-c, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, version=8, architecture=x86_64, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, vcs-type=git, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container) 2024-03-04T16:09:11.082 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:10 smithi083 podman[67215]: 2024-03-04 16:09:10.864397924 +0000 UTC m=+0.615230593 container cleanup e74b0ab73a802d289c44e66156695991f2d41618264fc9971a39b124060b467e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-c, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, release=754, RELEASE=HEAD, GIT_CLEAN=True, ceph=True, build-date=2022-05-03T08:36:31.336870, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, vcs-type=git, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., name=centos-stream) 2024-03-04T16:09:11.083 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:10 smithi083 bash[67215]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-c 2024-03-04T16:09:11.415 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 podman[67229]: 2024-03-04 16:09:11.082800974 +0000 UTC m=+0.524690029 container remove e74b0ab73a802d289c44e66156695991f2d41618264fc9971a39b124060b467e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-c, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., version=8, io.openshift.tags=base centos centos-stream, name=centos-stream, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, RELEASE=HEAD, GIT_CLEAN=True, release=754, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, distribution-scope=public, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-03-04T16:09:11.415 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-04T16:09:11.415 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 systemd[1]: Stopped Ceph mon.c for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:09:11.415 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mon.c.service: Consumed 8.260s CPU time. 2024-03-04T16:09:11.415 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 systemd[1]: Starting Ceph mon.c for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:09:11.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 podman[67363]: 2024-03-04 16:09:11.533774544 +0000 UTC m=+0.019162536 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:09:11.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 podman[67363]: 2024-03-04 16:09:11.658938421 +0000 UTC m=+0.144326454 container create eff1982ba275549fdc60ee853cf593c370f1fd818123e962b23527c0ccc34723 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-04T16:09:12.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 podman[67363]: 2024-03-04 16:09:11.922745229 +0000 UTC m=+0.408133227 container init eff1982ba275549fdc60ee853cf593c370f1fd818123e962b23527c0ccc34723 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:09:12.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 podman[67363]: 2024-03-04 16:09:11.925925112 +0000 UTC m=+0.411313108 container start eff1982ba275549fdc60ee853cf593c370f1fd818123e962b23527c0ccc34723 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-04T16:09:12.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: set uid:gid to 167:167 (ceph:ceph) 2024-03-04T16:09:12.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev), process ceph-mon, pid 2 2024-03-04T16:09:12.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: pidfile_write: ignore empty --pid-file 2024-03-04T16:09:12.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: load: jerasure load: lrc 2024-03-04T16:09:12.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: RocksDB version: 7.9.2 2024-03-04T16:09:12.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Git sha 0 2024-03-04T16:09:12.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Compile date 2024-03-04 11:37:16 2024-03-04T16:09:12.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: DB SUMMARY 2024-03-04T16:09:12.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: DB Session ID: R6UVLR8MU1HKCIGFNUGP 2024-03-04T16:09:12.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: CURRENT file: CURRENT 2024-03-04T16:09:12.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: IDENTITY file: IDENTITY 2024-03-04T16:09:12.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1084 Bytes 2024-03-04T16:09:12.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-03-04T16:09:12.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 629882 ; 2024-03-04T16:09:12.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.error_if_exists: 0 2024-03-04T16:09:12.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.create_if_missing: 0 2024-03-04T16:09:12.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.paranoid_checks: 1 2024-03-04T16:09:12.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-04T16:09:12.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-04T16:09:12.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-04T16:09:12.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.env: 0x5615bfe14b20 2024-03-04T16:09:12.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.fs: PosixFileSystem 2024-03-04T16:09:12.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.info_log: 0x5615c0f3b8e0 2024-03-04T16:09:12.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_file_opening_threads: 16 2024-03-04T16:09:12.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.statistics: (nil) 2024-03-04T16:09:12.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.use_fsync: 0 2024-03-04T16:09:12.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_log_file_size: 0 2024-03-04T16:09:12.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-04T16:09:12.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-04T16:09:12.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.keep_log_file_num: 1000 2024-03-04T16:09:12.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.recycle_log_file_num: 0 2024-03-04T16:09:12.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.allow_fallocate: 1 2024-03-04T16:09:12.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.allow_mmap_reads: 0 2024-03-04T16:09:12.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.allow_mmap_writes: 0 2024-03-04T16:09:12.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.use_direct_reads: 0 2024-03-04T16:09:12.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-04T16:09:12.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.create_missing_column_families: 0 2024-03-04T16:09:12.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.db_log_dir: 2024-03-04T16:09:12.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.wal_dir: 2024-03-04T16:09:12.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-04T16:09:12.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-04T16:09:12.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-04T16:09:12.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-04T16:09:12.048 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-04T16:09:12.048 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-04T16:09:12.048 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.advise_random_on_open: 1 2024-03-04T16:09:12.048 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.db_write_buffer_size: 0 2024-03-04T16:09:12.048 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.write_buffer_manager: 0x5615c0f0f400 2024-03-04T16:09:12.048 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-04T16:09:12.048 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-04T16:09:12.048 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-04T16:09:12.048 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.rate_limiter: (nil) 2024-03-04T16:09:12.048 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-04T16:09:12.048 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.wal_recovery_mode: 2 2024-03-04T16:09:12.048 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.enable_thread_tracking: 0 2024-03-04T16:09:12.048 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.enable_pipelined_write: 0 2024-03-04T16:09:12.048 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.unordered_write: 0 2024-03-04T16:09:12.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-04T16:09:12.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-04T16:09:12.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-04T16:09:12.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-04T16:09:12.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.row_cache: None 2024-03-04T16:09:12.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.wal_filter: None 2024-03-04T16:09:12.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-04T16:09:12.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.allow_ingest_behind: 0 2024-03-04T16:09:12.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.two_write_queues: 0 2024-03-04T16:09:12.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.manual_wal_flush: 0 2024-03-04T16:09:12.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.wal_compression: 0 2024-03-04T16:09:12.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.atomic_flush: 0 2024-03-04T16:09:12.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-04T16:09:12.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-04T16:09:12.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-04T16:09:12.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.log_readahead_size: 0 2024-03-04T16:09:12.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-04T16:09:12.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.best_efforts_recovery: 0 2024-03-04T16:09:12.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-04T16:09:12.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-04T16:09:12.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.allow_data_in_errors: 0 2024-03-04T16:09:12.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.db_host_id: __hostname__ 2024-03-04T16:09:12.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-04T16:09:12.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_background_jobs: 2 2024-03-04T16:09:12.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_background_compactions: -1 2024-03-04T16:09:12.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_subcompactions: 1 2024-03-04T16:09:12.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-04T16:09:12.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-04T16:09:12.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-04T16:09:12.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_total_wal_size: 0 2024-03-04T16:09:12.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-04T16:09:12.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-04T16:09:12.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-04T16:09:12.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-04T16:09:12.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_open_files: -1 2024-03-04T16:09:12.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.bytes_per_sync: 0 2024-03-04T16:09:12.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-04T16:09:12.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-04T16:09:12.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compaction_readahead_size: 0 2024-03-04T16:09:12.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_background_flushes: -1 2024-03-04T16:09:12.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Compression algorithms supported: 2024-03-04T16:09:12.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-04T16:09:12.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: kZSTD supported: 0 2024-03-04T16:09:12.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: kXpressCompression supported: 0 2024-03-04T16:09:12.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: kLZ4HCCompression supported: 1 2024-03-04T16:09:12.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: kZlibCompression supported: 1 2024-03-04T16:09:12.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: kSnappyCompression supported: 1 2024-03-04T16:09:12.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: kLZ4Compression supported: 1 2024-03-04T16:09:12.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: kBZip2Compression supported: 0 2024-03-04T16:09:12.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-04T16:09:12.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-04T16:09:12.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-03-04T16:09:12.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-04T16:09:12.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-04T16:09:12.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.merge_operator: 2024-03-04T16:09:12.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compaction_filter: None 2024-03-04T16:09:12.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compaction_filter_factory: None 2024-03-04T16:09:12.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.sst_partitioner_factory: None 2024-03-04T16:09:12.054 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-04T16:09:12.054 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-04T16:09:12.054 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5615c0f3b9e0) 2024-03-04T16:09:12.054 INFO:journalctl@ceph.mon.c.smithi083.stdout: cache_index_and_filter_blocks: 1 2024-03-04T16:09:12.054 INFO:journalctl@ceph.mon.c.smithi083.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-04T16:09:12.054 INFO:journalctl@ceph.mon.c.smithi083.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-04T16:09:12.054 INFO:journalctl@ceph.mon.c.smithi083.stdout: pin_top_level_index_and_filter: 1 2024-03-04T16:09:12.054 INFO:journalctl@ceph.mon.c.smithi083.stdout: index_type: 0 2024-03-04T16:09:12.054 INFO:journalctl@ceph.mon.c.smithi083.stdout: data_block_index_type: 0 2024-03-04T16:09:12.054 INFO:journalctl@ceph.mon.c.smithi083.stdout: index_shortening: 1 2024-03-04T16:09:12.054 INFO:journalctl@ceph.mon.c.smithi083.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-04T16:09:12.054 INFO:journalctl@ceph.mon.c.smithi083.stdout: checksum: 4 2024-03-04T16:09:12.054 INFO:journalctl@ceph.mon.c.smithi083.stdout: no_block_cache: 0 2024-03-04T16:09:12.054 INFO:journalctl@ceph.mon.c.smithi083.stdout: block_cache: 0x5615c0f41090 2024-03-04T16:09:12.055 INFO:journalctl@ceph.mon.c.smithi083.stdout: block_cache_name: BinnedLRUCache 2024-03-04T16:09:12.055 INFO:journalctl@ceph.mon.c.smithi083.stdout: block_cache_options: 2024-03-04T16:09:12.055 INFO:journalctl@ceph.mon.c.smithi083.stdout: capacity : 536870912 2024-03-04T16:09:12.055 INFO:journalctl@ceph.mon.c.smithi083.stdout: num_shard_bits : 4 2024-03-04T16:09:12.055 INFO:journalctl@ceph.mon.c.smithi083.stdout: strict_capacity_limit : 0 2024-03-04T16:09:12.055 INFO:journalctl@ceph.mon.c.smithi083.stdout: high_pri_pool_ratio: 0.000 2024-03-04T16:09:12.055 INFO:journalctl@ceph.mon.c.smithi083.stdout: block_cache_compressed: (nil) 2024-03-04T16:09:12.055 INFO:journalctl@ceph.mon.c.smithi083.stdout: persistent_cache: (nil) 2024-03-04T16:09:12.055 INFO:journalctl@ceph.mon.c.smithi083.stdout: block_size: 4096 2024-03-04T16:09:12.055 INFO:journalctl@ceph.mon.c.smithi083.stdout: block_size_deviation: 10 2024-03-04T16:09:12.055 INFO:journalctl@ceph.mon.c.smithi083.stdout: block_restart_interval: 16 2024-03-04T16:09:12.055 INFO:journalctl@ceph.mon.c.smithi083.stdout: index_block_restart_interval: 1 2024-03-04T16:09:12.055 INFO:journalctl@ceph.mon.c.smithi083.stdout: metadata_block_size: 4096 2024-03-04T16:09:12.055 INFO:journalctl@ceph.mon.c.smithi083.stdout: partition_filters: 0 2024-03-04T16:09:12.055 INFO:journalctl@ceph.mon.c.smithi083.stdout: use_delta_encoding: 1 2024-03-04T16:09:12.055 INFO:journalctl@ceph.mon.c.smithi083.stdout: filter_policy: bloomfilter 2024-03-04T16:09:12.056 INFO:journalctl@ceph.mon.c.smithi083.stdout: whole_key_filtering: 1 2024-03-04T16:09:12.056 INFO:journalctl@ceph.mon.c.smithi083.stdout: verify_compression: 0 2024-03-04T16:09:12.056 INFO:journalctl@ceph.mon.c.smithi083.stdout: read_amp_bytes_per_bit: 0 2024-03-04T16:09:12.056 INFO:journalctl@ceph.mon.c.smithi083.stdout: format_version: 5 2024-03-04T16:09:12.056 INFO:journalctl@ceph.mon.c.smithi083.stdout: enable_index_compression: 1 2024-03-04T16:09:12.056 INFO:journalctl@ceph.mon.c.smithi083.stdout: block_align: 0 2024-03-04T16:09:12.056 INFO:journalctl@ceph.mon.c.smithi083.stdout: max_auto_readahead_size: 262144 2024-03-04T16:09:12.056 INFO:journalctl@ceph.mon.c.smithi083.stdout: prepopulate_block_cache: 0 2024-03-04T16:09:12.056 INFO:journalctl@ceph.mon.c.smithi083.stdout: initial_auto_readahead_size: 8192 2024-03-04T16:09:12.056 INFO:journalctl@ceph.mon.c.smithi083.stdout: num_file_reads_for_auto_readahead: 2 2024-03-04T16:09:12.056 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.write_buffer_size: 33554432 2024-03-04T16:09:12.056 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_write_buffer_number: 2 2024-03-04T16:09:12.056 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compression: NoCompression 2024-03-04T16:09:12.056 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.bottommost_compression: Disabled 2024-03-04T16:09:12.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.prefix_extractor: nullptr 2024-03-04T16:09:12.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-04T16:09:12.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.num_levels: 7 2024-03-04T16:09:12.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-04T16:09:12.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-04T16:09:12.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-04T16:09:12.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-04T16:09:12.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-04T16:09:12.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-04T16:09:12.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-04T16:09:12.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-04T16:09:12.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-04T16:09:12.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-04T16:09:12.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-04T16:09:12.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-04T16:09:12.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-04T16:09:12.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compression_opts.level: 32767 2024-03-04T16:09:12.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compression_opts.strategy: 0 2024-03-04T16:09:12.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-04T16:09:12.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-04T16:09:12.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-04T16:09:12.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-04T16:09:12.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compression_opts.enabled: false 2024-03-04T16:09:12.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-04T16:09:12.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-04T16:09:12.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-04T16:09:12.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-04T16:09:12.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.target_file_size_base: 67108864 2024-03-04T16:09:12.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-04T16:09:12.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-04T16:09:12.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-04T16:09:12.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-04T16:09:12.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-04T16:09:12.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-04T16:09:12.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-04T16:09:12.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-04T16:09:12.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-04T16:09:12.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-04T16:09:12.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-04T16:09:12.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-04T16:09:12.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-04T16:09:12.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-04T16:09:12.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.arena_block_size: 1048576 2024-03-04T16:09:12.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-04T16:09:12.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-04T16:09:12.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.disable_auto_compactions: 0 2024-03-04T16:09:12.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-04T16:09:12.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-04T16:09:12.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-04T16:09:12.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-04T16:09:12.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-04T16:09:12.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-04T16:09:12.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-04T16:09:12.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-04T16:09:12.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-04T16:09:12.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-04T16:09:12.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.table_properties_collectors: 2024-03-04T16:09:12.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.inplace_update_support: 0 2024-03-04T16:09:12.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-04T16:09:12.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-04T16:09:12.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-04T16:09:12.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-04T16:09:12.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.bloom_locality: 0 2024-03-04T16:09:12.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.max_successive_merges: 0 2024-03-04T16:09:12.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-04T16:09:12.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.paranoid_file_checks: 0 2024-03-04T16:09:12.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.force_consistency_checks: 1 2024-03-04T16:09:12.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.report_bg_io_stats: 0 2024-03-04T16:09:12.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.ttl: 2592000 2024-03-04T16:09:12.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-04T16:09:12.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-04T16:09:12.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-04T16:09:12.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.enable_blob_files: false 2024-03-04T16:09:12.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.min_blob_size: 0 2024-03-04T16:09:12.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.blob_file_size: 268435456 2024-03-04T16:09:12.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-04T16:09:12.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-04T16:09:12.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-04T16:09:12.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-04T16:09:12.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-04T16:09:12.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.blob_file_starting_level: 0 2024-03-04T16:09:12.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-04T16:09:12.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-03-04T16:09:12.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-04T16:09:12.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 15168, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-04T16:09:12.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-03-04T16:09:12.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 934b6614-12a4-480b-9db2-ae1ee31c7c1a 2024-03-04T16:09:12.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709568551955390, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-03-04T16:09:12.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-03-04T16:09:12.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709568551957557, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 377163, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 15173, "largest_seqno": 15386, "table_properties": {"data_size": 375070, "index_size": 761, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 325, "raw_key_size": 2592, "raw_average_key_size": 24, "raw_value_size": 372582, "raw_average_value_size": 3582, "num_data_blocks": 33, "num_entries": 104, "num_filter_entries": 104, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709568551, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "934b6614-12a4-480b-9db2-ae1ee31c7c1a", "db_session_id": "R6UVLR8MU1HKCIGFNUGP", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-03-04T16:09:12.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709568551957639, "job": 1, "event": "recovery_finished"} 2024-03-04T16:09:12.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-03-04T16:09:12.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-04T16:09:12.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-04T16:09:12.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5615c1034000 2024-03-04T16:09:12.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: DB pointer 0x5615c101e000 2024-03-04T16:09:12.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-04T16:09:12.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-04T16:09:12.064 INFO:journalctl@ceph.mon.c.smithi083.stdout: ** DB Stats ** 2024-03-04T16:09:12.064 INFO:journalctl@ceph.mon.c.smithi083.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T16:09:12.064 INFO:journalctl@ceph.mon.c.smithi083.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-04T16:09:12.065 INFO:journalctl@ceph.mon.c.smithi083.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T16:09:12.065 INFO:journalctl@ceph.mon.c.smithi083.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T16:09:12.065 INFO:journalctl@ceph.mon.c.smithi083.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-04T16:09:12.065 INFO:journalctl@ceph.mon.c.smithi083.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T16:09:12.065 INFO:journalctl@ceph.mon.c.smithi083.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T16:09:12.065 INFO:journalctl@ceph.mon.c.smithi083.stdout: 2024-03-04T16:09:12.065 INFO:journalctl@ceph.mon.c.smithi083.stdout: ** Compaction Stats [default] ** 2024-03-04T16:09:12.065 INFO:journalctl@ceph.mon.c.smithi083.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T16:09:12.065 INFO:journalctl@ceph.mon.c.smithi083.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-04T16:09:12.065 INFO:journalctl@ceph.mon.c.smithi083.stdout: L0 1/0 368.32 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 288.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T16:09:12.065 INFO:journalctl@ceph.mon.c.smithi083.stdout: L6 1/0 11.19 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-04T16:09:12.065 INFO:journalctl@ceph.mon.c.smithi083.stdout: Sum 2/0 11.55 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 288.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T16:09:12.065 INFO:journalctl@ceph.mon.c.smithi083.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 288.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T16:09:12.065 INFO:journalctl@ceph.mon.c.smithi083.stdout: 2024-03-04T16:09:12.066 INFO:journalctl@ceph.mon.c.smithi083.stdout: ** Compaction Stats [default] ** 2024-03-04T16:09:12.066 INFO:journalctl@ceph.mon.c.smithi083.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T16:09:12.066 INFO:journalctl@ceph.mon.c.smithi083.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T16:09:12.066 INFO:journalctl@ceph.mon.c.smithi083.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 288.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T16:09:12.066 INFO:journalctl@ceph.mon.c.smithi083.stdout: 2024-03-04T16:09:12.066 INFO:journalctl@ceph.mon.c.smithi083.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-04T16:09:12.066 INFO:journalctl@ceph.mon.c.smithi083.stdout: 2024-03-04T16:09:12.066 INFO:journalctl@ceph.mon.c.smithi083.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T16:09:12.066 INFO:journalctl@ceph.mon.c.smithi083.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-04T16:09:12.066 INFO:journalctl@ceph.mon.c.smithi083.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T16:09:12.066 INFO:journalctl@ceph.mon.c.smithi083.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T16:09:12.066 INFO:journalctl@ceph.mon.c.smithi083.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T16:09:12.066 INFO:journalctl@ceph.mon.c.smithi083.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T16:09:12.066 INFO:journalctl@ceph.mon.c.smithi083.stdout: Cumulative compaction: 0.00 GB write, 85.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T16:09:12.066 INFO:journalctl@ceph.mon.c.smithi083.stdout: Interval compaction: 0.00 GB write, 85.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T16:09:12.067 INFO:journalctl@ceph.mon.c.smithi083.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T16:09:12.067 INFO:journalctl@ceph.mon.c.smithi083.stdout: Block cache BinnedLRUCache@0x5615c0f41090#2 capacity: 512.00 MB usage: 1.20 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-04T16:09:12.067 INFO:journalctl@ceph.mon.c.smithi083.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.38 KB,7.15256e-05%) IndexBlock(1,0.83 KB,0.000157952%) Misc(1,0.00 KB,0%) 2024-03-04T16:09:12.067 INFO:journalctl@ceph.mon.c.smithi083.stdout: 2024-03-04T16:09:12.067 INFO:journalctl@ceph.mon.c.smithi083.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T16:09:12.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: starting mon.c rank 1 at public addrs [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] at bind addrs [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T16:09:12.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: mon.c@-1(???) e3 preinit fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T16:09:12.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: mon.c@-1(???).mds e1 new map 2024-03-04T16:09:12.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: mon.c@-1(???).mds e1 print_map 2024-03-04T16:09:12.067 INFO:journalctl@ceph.mon.c.smithi083.stdout: e1 2024-03-04T16:09:12.067 INFO:journalctl@ceph.mon.c.smithi083.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:09:12.067 INFO:journalctl@ceph.mon.c.smithi083.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:09:12.067 INFO:journalctl@ceph.mon.c.smithi083.stdout: legacy client fscid: -1 2024-03-04T16:09:12.067 INFO:journalctl@ceph.mon.c.smithi083.stdout: 2024-03-04T16:09:12.068 INFO:journalctl@ceph.mon.c.smithi083.stdout: No filesystems configured 2024-03-04T16:09:12.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: mon.c@-1(???).osd e106 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-04T16:09:12.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-04T16:09:12.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-04T16:09:12.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-04T16:09:12.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:11 smithi083 ceph-mon[67383]: mon.c@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-03-04T16:09:12.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:12 smithi083 bash[67363]: eff1982ba275549fdc60ee853cf593c370f1fd818123e962b23527c0ccc34723 2024-03-04T16:09:12.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:12 smithi083 systemd[1]: Started Ceph mon.c for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:09:13.359 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[64844]: mon.c calling monitor election 2024-03-04T16:09:13.359 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[64844]: mon.a calling monitor election 2024-03-04T16:09:13.359 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[64844]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T16:09:13.359 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[64844]: monmap e3: 3 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],b=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],c=[v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T16:09:13.359 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[64844]: fsmap 2024-03-04T16:09:13.360 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[64844]: osdmap e106: 8 total, 8 up, 8 in 2024-03-04T16:09:13.360 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[64844]: mgrmap e37: y(active, since 15s), standbys: x 2024-03-04T16:09:13.360 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[64844]: overall HEALTH_OK 2024-03-04T16:09:13.360 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:13.360 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:13.360 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:13.360 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:13.360 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[64844]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:09:13.360 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[67383]: mon.c calling monitor election 2024-03-04T16:09:13.360 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[67383]: mon.a calling monitor election 2024-03-04T16:09:13.360 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[67383]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T16:09:13.361 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[67383]: monmap e3: 3 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],b=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],c=[v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T16:09:13.361 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[67383]: fsmap 2024-03-04T16:09:13.361 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[67383]: osdmap e106: 8 total, 8 up, 8 in 2024-03-04T16:09:13.361 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[67383]: mgrmap e37: y(active, since 15s), standbys: x 2024-03-04T16:09:13.361 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[67383]: overall HEALTH_OK 2024-03-04T16:09:13.361 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:13.361 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:13.361 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:13.361 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:13.361 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:13 smithi083 ceph-mon[67383]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:09:13.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:13 smithi086 ceph-mon[25584]: mon.c calling monitor election 2024-03-04T16:09:13.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:13 smithi086 ceph-mon[25584]: mon.a calling monitor election 2024-03-04T16:09:13.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:13 smithi086 ceph-mon[25584]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T16:09:13.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:13 smithi086 ceph-mon[25584]: monmap e3: 3 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],b=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],c=[v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T16:09:13.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:13 smithi086 ceph-mon[25584]: fsmap 2024-03-04T16:09:13.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:13 smithi086 ceph-mon[25584]: osdmap e106: 8 total, 8 up, 8 in 2024-03-04T16:09:13.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:13 smithi086 ceph-mon[25584]: mgrmap e37: y(active, since 15s), standbys: x 2024-03-04T16:09:13.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:13 smithi086 ceph-mon[25584]: overall HEALTH_OK 2024-03-04T16:09:13.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:13 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:13.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:13 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:13.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:13 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:13.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:13 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:13.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:13 smithi086 ceph-mon[25584]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:09:15.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:15 smithi083 ceph-mon[64844]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:09:15.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:15 smithi083 ceph-mon[67383]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:09:15.989 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:09:15 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:09:15] "GET /metrics HTTP/1.1" 200 38181 "" "Prometheus/2.33.4" 2024-03-04T16:09:16.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:15 smithi086 ceph-mon[25584]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T16:09:17.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:09:16 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:09:16.965+0000 7fe862dc1700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-04T16:09:17.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:17 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:17.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:17 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:17.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:17 smithi086 ceph-mon[25584]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:17.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:17 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:17.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:17 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:17.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:17 smithi083 ceph-mon[67383]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:17.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:17 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:17.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:17 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:17.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:17 smithi083 ceph-mon[64844]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:18.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:18 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:18.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:18 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:18.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:18 smithi086 ceph-mon[25584]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:18.985 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:09:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:09:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:09:18.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:18 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:18.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:18 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:18.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:18 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:18.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:18 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:18.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:18 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:18.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:18 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:19.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:19 smithi086 ceph-mon[25584]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:19.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:19 smithi083 ceph-mon[64844]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:19.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:19 smithi083 ceph-mon[67383]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:22.153 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:21 smithi086 ceph-mon[25584]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:22.153 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:21 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:22.153 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:21 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:22.153 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:21 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:22.153 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:21 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:09:22.153 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:21 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:22.153 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:21 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:09:22.153 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:21 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:22.154 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:21 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-04T16:09:22.154 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:21 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-04T16:09:22.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[64844]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:22.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:22.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:22.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:22.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:09:22.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:22.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:09:22.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:22.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-04T16:09:22.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-04T16:09:22.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[67383]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:22.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:22.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:22.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:22.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:09:22.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:22.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:09:22.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:22.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-04T16:09:22.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:21 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-04T16:09:23.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:22 smithi086 ceph-mon[25584]: Upgrade: It appears safe to stop mon.b 2024-03-04T16:09:23.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:22 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:23.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:22 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:09:23.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:22 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:09:23.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:22 smithi086 ceph-mon[25584]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:23.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:22 smithi083 ceph-mon[67383]: Upgrade: It appears safe to stop mon.b 2024-03-04T16:09:23.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:22 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:23.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:22 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:09:23.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:22 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:09:23.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:22 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:23.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:22 smithi083 ceph-mon[64844]: Upgrade: It appears safe to stop mon.b 2024-03-04T16:09:23.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:22 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:23.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:22 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:09:23.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:22 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:09:23.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:22 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:23.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:23 smithi086 ceph-mon[25584]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:23.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:23 smithi086 ceph-mon[25584]: Upgrade: Updating mon.b 2024-03-04T16:09:23.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:23 smithi086 ceph-mon[25584]: Deploying daemon mon.b on smithi086 2024-03-04T16:09:24.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:23 smithi083 ceph-mon[67383]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:24.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:23 smithi083 ceph-mon[67383]: Upgrade: Updating mon.b 2024-03-04T16:09:24.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:23 smithi083 ceph-mon[67383]: Deploying daemon mon.b on smithi086 2024-03-04T16:09:24.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:23 smithi083 ceph-mon[64844]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:24.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:23 smithi083 ceph-mon[64844]: Upgrade: Updating mon.b 2024-03-04T16:09:24.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:23 smithi083 ceph-mon[64844]: Deploying daemon mon.b on smithi086 2024-03-04T16:09:24.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:24 smithi086 systemd[1]: Stopping Ceph mon.b for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:09:24.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:24 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-b[25580]: 2024-03-04T16:09:24.883+0000 7fdd5103a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:09:24.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:24 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-b[25580]: 2024-03-04T16:09:24.883+0000 7fdd5103a700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-04T16:09:25.268 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:24 smithi086 podman[57755]: 2024-03-04 16:09:24.986466014 +0000 UTC m=+0.198029447 container died d1b972ebdd2a1d345a122b33a9d2e3240b5d6521f265ad809fbd9031c0bd3ddd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-b, vendor=Red Hat, Inc., vcs-type=git, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, ceph=True, release=754, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux ) 2024-03-04T16:09:25.652 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:25 smithi086 podman[57755]: 2024-03-04 16:09:25.367391108 +0000 UTC m=+0.578954543 container cleanup d1b972ebdd2a1d345a122b33a9d2e3240b5d6521f265ad809fbd9031c0bd3ddd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-b, name=centos-stream, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, distribution-scope=public, maintainer=Guillaume Abrioux , RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, vcs-type=git, io.buildah.version=1.19.8, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-04T16:09:25.652 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:25 smithi086 bash[57755]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-b 2024-03-04T16:09:25.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:25 smithi086 podman[57769]: 2024-03-04 16:09:25.65202365 +0000 UTC m=+0.660680863 container remove d1b972ebdd2a1d345a122b33a9d2e3240b5d6521f265ad809fbd9031c0bd3ddd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-b, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, version=8, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, distribution-scope=public, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, ceph=True, GIT_BRANCH=HEAD, RELEASE=HEAD) 2024-03-04T16:09:25.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:25 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-04T16:09:25.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:25 smithi086 systemd[1]: Stopped Ceph mon.b for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:09:25.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:25 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mon.b.service: Consumed 9.541s CPU time. 2024-03-04T16:09:25.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:09:25 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:09:25] "GET /metrics HTTP/1.1" 200 38181 "" "Prometheus/2.33.4" 2024-03-04T16:09:26.321 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:25 smithi086 systemd[1]: Starting Ceph mon.b for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:09:26.321 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 podman[57900]: 2024-03-04 16:09:26.10229686 +0000 UTC m=+0.018068356 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:09:26.321 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 podman[57900]: 2024-03-04 16:09:26.211329271 +0000 UTC m=+0.127100797 container create 71557985ed5ad283b9dfab138ceafb6c19157775a91ae47f2830131208faa637 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:09:26.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 podman[57900]: 2024-03-04 16:09:26.430267233 +0000 UTC m=+0.346038728 container init 71557985ed5ad283b9dfab138ceafb6c19157775a91ae47f2830131208faa637 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:09:26.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 podman[57900]: 2024-03-04 16:09:26.43350627 +0000 UTC m=+0.349277753 container start 71557985ed5ad283b9dfab138ceafb6c19157775a91ae47f2830131208faa637 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, ceph=True) 2024-03-04T16:09:26.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: set uid:gid to 167:167 (ceph:ceph) 2024-03-04T16:09:26.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev), process ceph-mon, pid 2 2024-03-04T16:09:26.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: pidfile_write: ignore empty --pid-file 2024-03-04T16:09:26.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: load: jerasure load: lrc 2024-03-04T16:09:26.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: RocksDB version: 7.9.2 2024-03-04T16:09:26.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Git sha 0 2024-03-04T16:09:26.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Compile date 2024-03-04 11:37:16 2024-03-04T16:09:26.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: DB SUMMARY 2024-03-04T16:09:26.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: DB Session ID: EUKZXVJ2GFG47MG7L7FL 2024-03-04T16:09:26.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: CURRENT file: CURRENT 2024-03-04T16:09:26.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: IDENTITY file: IDENTITY 2024-03-04T16:09:26.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1084 Bytes 2024-03-04T16:09:26.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-03-04T16:09:26.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 1369057 ; 2024-03-04T16:09:26.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.error_if_exists: 0 2024-03-04T16:09:26.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.create_if_missing: 0 2024-03-04T16:09:26.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.paranoid_checks: 1 2024-03-04T16:09:26.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-04T16:09:26.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-04T16:09:26.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-04T16:09:26.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.env: 0x5640b5eecb20 2024-03-04T16:09:26.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.fs: PosixFileSystem 2024-03-04T16:09:26.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.info_log: 0x5640b6ced8e0 2024-03-04T16:09:26.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_file_opening_threads: 16 2024-03-04T16:09:26.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.statistics: (nil) 2024-03-04T16:09:26.738 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.use_fsync: 0 2024-03-04T16:09:26.738 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_log_file_size: 0 2024-03-04T16:09:26.738 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-04T16:09:26.738 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-04T16:09:26.738 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.keep_log_file_num: 1000 2024-03-04T16:09:26.738 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.recycle_log_file_num: 0 2024-03-04T16:09:26.738 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.allow_fallocate: 1 2024-03-04T16:09:26.738 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.allow_mmap_reads: 0 2024-03-04T16:09:26.738 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.allow_mmap_writes: 0 2024-03-04T16:09:26.738 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.use_direct_reads: 0 2024-03-04T16:09:26.738 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-04T16:09:26.739 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.create_missing_column_families: 0 2024-03-04T16:09:26.739 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.db_log_dir: 2024-03-04T16:09:26.739 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.wal_dir: 2024-03-04T16:09:26.739 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-04T16:09:26.739 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-04T16:09:26.739 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-04T16:09:26.739 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-04T16:09:26.739 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-04T16:09:26.739 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-04T16:09:26.739 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.advise_random_on_open: 1 2024-03-04T16:09:26.739 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.db_write_buffer_size: 0 2024-03-04T16:09:26.739 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.write_buffer_manager: 0x5640b6cc1400 2024-03-04T16:09:26.739 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-04T16:09:26.739 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-04T16:09:26.739 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-04T16:09:26.740 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.rate_limiter: (nil) 2024-03-04T16:09:26.740 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-04T16:09:26.740 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.wal_recovery_mode: 2 2024-03-04T16:09:26.740 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.enable_thread_tracking: 0 2024-03-04T16:09:26.740 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.enable_pipelined_write: 0 2024-03-04T16:09:26.740 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.unordered_write: 0 2024-03-04T16:09:26.741 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-04T16:09:26.741 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-04T16:09:26.741 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-04T16:09:26.741 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-04T16:09:26.741 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.row_cache: None 2024-03-04T16:09:26.741 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.wal_filter: None 2024-03-04T16:09:26.741 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-04T16:09:26.741 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.allow_ingest_behind: 0 2024-03-04T16:09:26.741 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.two_write_queues: 0 2024-03-04T16:09:26.741 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.manual_wal_flush: 0 2024-03-04T16:09:26.741 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.wal_compression: 0 2024-03-04T16:09:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.atomic_flush: 0 2024-03-04T16:09:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-04T16:09:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-04T16:09:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-04T16:09:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.log_readahead_size: 0 2024-03-04T16:09:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-04T16:09:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.best_efforts_recovery: 0 2024-03-04T16:09:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-04T16:09:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-04T16:09:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.allow_data_in_errors: 0 2024-03-04T16:09:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.db_host_id: __hostname__ 2024-03-04T16:09:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-04T16:09:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_background_jobs: 2 2024-03-04T16:09:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_background_compactions: -1 2024-03-04T16:09:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_subcompactions: 1 2024-03-04T16:09:26.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-04T16:09:26.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-04T16:09:26.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-04T16:09:26.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_total_wal_size: 0 2024-03-04T16:09:26.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-04T16:09:26.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-04T16:09:26.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-04T16:09:26.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-04T16:09:26.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_open_files: -1 2024-03-04T16:09:26.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.bytes_per_sync: 0 2024-03-04T16:09:26.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-04T16:09:26.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-04T16:09:26.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compaction_readahead_size: 0 2024-03-04T16:09:26.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_background_flushes: -1 2024-03-04T16:09:26.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Compression algorithms supported: 2024-03-04T16:09:26.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-04T16:09:26.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: kZSTD supported: 0 2024-03-04T16:09:26.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: kXpressCompression supported: 0 2024-03-04T16:09:26.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: kLZ4HCCompression supported: 1 2024-03-04T16:09:26.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: kZlibCompression supported: 1 2024-03-04T16:09:26.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: kSnappyCompression supported: 1 2024-03-04T16:09:26.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: kLZ4Compression supported: 1 2024-03-04T16:09:26.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: kBZip2Compression supported: 0 2024-03-04T16:09:26.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-04T16:09:26.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-04T16:09:26.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-03-04T16:09:26.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-04T16:09:26.745 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-04T16:09:26.745 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.merge_operator: 2024-03-04T16:09:26.745 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compaction_filter: None 2024-03-04T16:09:26.745 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compaction_filter_factory: None 2024-03-04T16:09:26.745 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.sst_partitioner_factory: None 2024-03-04T16:09:26.745 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-04T16:09:26.745 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-04T16:09:26.745 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5640b6ced9e0) 2024-03-04T16:09:26.745 INFO:journalctl@ceph.mon.b.smithi086.stdout: cache_index_and_filter_blocks: 1 2024-03-04T16:09:26.745 INFO:journalctl@ceph.mon.b.smithi086.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-04T16:09:26.745 INFO:journalctl@ceph.mon.b.smithi086.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-04T16:09:26.745 INFO:journalctl@ceph.mon.b.smithi086.stdout: pin_top_level_index_and_filter: 1 2024-03-04T16:09:26.745 INFO:journalctl@ceph.mon.b.smithi086.stdout: index_type: 0 2024-03-04T16:09:26.746 INFO:journalctl@ceph.mon.b.smithi086.stdout: data_block_index_type: 0 2024-03-04T16:09:26.746 INFO:journalctl@ceph.mon.b.smithi086.stdout: index_shortening: 1 2024-03-04T16:09:26.746 INFO:journalctl@ceph.mon.b.smithi086.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-04T16:09:26.746 INFO:journalctl@ceph.mon.b.smithi086.stdout: checksum: 4 2024-03-04T16:09:26.746 INFO:journalctl@ceph.mon.b.smithi086.stdout: no_block_cache: 0 2024-03-04T16:09:26.746 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_cache: 0x5640b6cf3090 2024-03-04T16:09:26.746 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_cache_name: BinnedLRUCache 2024-03-04T16:09:26.746 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_cache_options: 2024-03-04T16:09:26.746 INFO:journalctl@ceph.mon.b.smithi086.stdout: capacity : 536870912 2024-03-04T16:09:26.746 INFO:journalctl@ceph.mon.b.smithi086.stdout: num_shard_bits : 4 2024-03-04T16:09:26.746 INFO:journalctl@ceph.mon.b.smithi086.stdout: strict_capacity_limit : 0 2024-03-04T16:09:26.746 INFO:journalctl@ceph.mon.b.smithi086.stdout: high_pri_pool_ratio: 0.000 2024-03-04T16:09:26.746 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_cache_compressed: (nil) 2024-03-04T16:09:26.746 INFO:journalctl@ceph.mon.b.smithi086.stdout: persistent_cache: (nil) 2024-03-04T16:09:26.746 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_size: 4096 2024-03-04T16:09:26.747 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_size_deviation: 10 2024-03-04T16:09:26.747 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_restart_interval: 16 2024-03-04T16:09:26.747 INFO:journalctl@ceph.mon.b.smithi086.stdout: index_block_restart_interval: 1 2024-03-04T16:09:26.747 INFO:journalctl@ceph.mon.b.smithi086.stdout: metadata_block_size: 4096 2024-03-04T16:09:26.747 INFO:journalctl@ceph.mon.b.smithi086.stdout: partition_filters: 0 2024-03-04T16:09:26.747 INFO:journalctl@ceph.mon.b.smithi086.stdout: use_delta_encoding: 1 2024-03-04T16:09:26.747 INFO:journalctl@ceph.mon.b.smithi086.stdout: filter_policy: bloomfilter 2024-03-04T16:09:26.747 INFO:journalctl@ceph.mon.b.smithi086.stdout: whole_key_filtering: 1 2024-03-04T16:09:26.747 INFO:journalctl@ceph.mon.b.smithi086.stdout: verify_compression: 0 2024-03-04T16:09:26.747 INFO:journalctl@ceph.mon.b.smithi086.stdout: read_amp_bytes_per_bit: 0 2024-03-04T16:09:26.747 INFO:journalctl@ceph.mon.b.smithi086.stdout: format_version: 5 2024-03-04T16:09:26.747 INFO:journalctl@ceph.mon.b.smithi086.stdout: enable_index_compression: 1 2024-03-04T16:09:26.747 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_align: 0 2024-03-04T16:09:26.747 INFO:journalctl@ceph.mon.b.smithi086.stdout: max_auto_readahead_size: 262144 2024-03-04T16:09:26.748 INFO:journalctl@ceph.mon.b.smithi086.stdout: prepopulate_block_cache: 0 2024-03-04T16:09:26.748 INFO:journalctl@ceph.mon.b.smithi086.stdout: initial_auto_readahead_size: 8192 2024-03-04T16:09:26.748 INFO:journalctl@ceph.mon.b.smithi086.stdout: num_file_reads_for_auto_readahead: 2 2024-03-04T16:09:26.748 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.write_buffer_size: 33554432 2024-03-04T16:09:26.748 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_write_buffer_number: 2 2024-03-04T16:09:26.748 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compression: NoCompression 2024-03-04T16:09:26.748 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.bottommost_compression: Disabled 2024-03-04T16:09:26.748 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.prefix_extractor: nullptr 2024-03-04T16:09:26.748 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-04T16:09:26.748 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.num_levels: 7 2024-03-04T16:09:26.748 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-04T16:09:26.748 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-04T16:09:26.748 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-04T16:09:26.748 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-04T16:09:26.749 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-04T16:09:26.749 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-04T16:09:26.749 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-04T16:09:26.749 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-04T16:09:26.749 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-04T16:09:26.749 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-04T16:09:26.749 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-04T16:09:26.749 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-04T16:09:26.749 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-04T16:09:26.749 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compression_opts.level: 32767 2024-03-04T16:09:26.749 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compression_opts.strategy: 0 2024-03-04T16:09:26.749 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-04T16:09:26.749 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-04T16:09:26.749 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-04T16:09:26.750 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-04T16:09:26.750 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compression_opts.enabled: false 2024-03-04T16:09:26.750 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-04T16:09:26.750 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-04T16:09:26.750 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-04T16:09:26.750 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-04T16:09:26.750 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.target_file_size_base: 67108864 2024-03-04T16:09:26.750 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-04T16:09:26.750 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-04T16:09:26.750 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-04T16:09:26.750 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-04T16:09:26.750 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-04T16:09:26.750 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-04T16:09:26.750 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-04T16:09:26.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-04T16:09:26.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-04T16:09:26.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-04T16:09:26.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-04T16:09:26.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-04T16:09:26.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-04T16:09:26.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-04T16:09:26.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.arena_block_size: 1048576 2024-03-04T16:09:26.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-04T16:09:26.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-04T16:09:26.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.disable_auto_compactions: 0 2024-03-04T16:09:26.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-04T16:09:26.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-04T16:09:26.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-04T16:09:26.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-04T16:09:26.752 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-04T16:09:26.752 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-04T16:09:26.752 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-04T16:09:26.752 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-04T16:09:26.752 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-04T16:09:26.752 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-04T16:09:26.752 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.table_properties_collectors: 2024-03-04T16:09:26.752 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.inplace_update_support: 0 2024-03-04T16:09:26.752 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-04T16:09:26.752 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-04T16:09:26.752 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-04T16:09:26.752 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-04T16:09:26.752 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.bloom_locality: 0 2024-03-04T16:09:26.752 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.max_successive_merges: 0 2024-03-04T16:09:26.753 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-04T16:09:26.753 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.paranoid_file_checks: 0 2024-03-04T16:09:26.753 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.force_consistency_checks: 1 2024-03-04T16:09:26.753 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.report_bg_io_stats: 0 2024-03-04T16:09:26.753 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.ttl: 2592000 2024-03-04T16:09:26.753 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-04T16:09:26.753 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-04T16:09:26.753 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-04T16:09:26.753 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.enable_blob_files: false 2024-03-04T16:09:26.753 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.min_blob_size: 0 2024-03-04T16:09:26.753 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.blob_file_size: 268435456 2024-03-04T16:09:26.753 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-04T16:09:26.753 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-04T16:09:26.753 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-04T16:09:26.753 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-04T16:09:26.754 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-04T16:09:26.754 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.blob_file_starting_level: 0 2024-03-04T16:09:26.754 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-04T16:09:26.754 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-03-04T16:09:26.754 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-04T16:09:26.754 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 15069, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-04T16:09:26.754 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-03-04T16:09:26.754 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 690b338e-ae12-4db7-bbef-272da46febef 2024-03-04T16:09:26.754 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709568566462679, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-03-04T16:09:26.754 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-03-04T16:09:26.754 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709568566466667, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 799775, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 15074, "largest_seqno": 15811, "table_properties": {"data_size": 796031, "index_size": 1513, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1221, "raw_key_size": 9303, "raw_average_key_size": 20, "raw_value_size": 787963, "raw_average_value_size": 1709, "num_data_blocks": 69, "num_entries": 461, "num_filter_entries": 461, "num_deletions": 251, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709568566, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "690b338e-ae12-4db7-bbef-272da46febef", "db_session_id": "EUKZXVJ2GFG47MG7L7FL", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-03-04T16:09:26.754 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709568566466759, "job": 1, "event": "recovery_finished"} 2024-03-04T16:09:26.754 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-03-04T16:09:26.755 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-04T16:09:26.755 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-04T16:09:26.755 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5640b6de6000 2024-03-04T16:09:26.755 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: DB pointer 0x5640b6dd0000 2024-03-04T16:09:26.755 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-04T16:09:26.755 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-04T16:09:26.755 INFO:journalctl@ceph.mon.b.smithi086.stdout: ** DB Stats ** 2024-03-04T16:09:26.755 INFO:journalctl@ceph.mon.b.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T16:09:26.755 INFO:journalctl@ceph.mon.b.smithi086.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-04T16:09:26.755 INFO:journalctl@ceph.mon.b.smithi086.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T16:09:26.755 INFO:journalctl@ceph.mon.b.smithi086.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T16:09:26.755 INFO:journalctl@ceph.mon.b.smithi086.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-04T16:09:26.755 INFO:journalctl@ceph.mon.b.smithi086.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T16:09:26.755 INFO:journalctl@ceph.mon.b.smithi086.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T16:09:26.756 INFO:journalctl@ceph.mon.b.smithi086.stdout: 2024-03-04T16:09:26.756 INFO:journalctl@ceph.mon.b.smithi086.stdout: ** Compaction Stats [default] ** 2024-03-04T16:09:26.756 INFO:journalctl@ceph.mon.b.smithi086.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T16:09:26.756 INFO:journalctl@ceph.mon.b.smithi086.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-04T16:09:26.756 INFO:journalctl@ceph.mon.b.smithi086.stdout: L0 1/0 781.03 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 371.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-04T16:09:26.756 INFO:journalctl@ceph.mon.b.smithi086.stdout: L6 1/0 11.19 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-04T16:09:26.756 INFO:journalctl@ceph.mon.b.smithi086.stdout: Sum 2/0 11.96 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 371.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-04T16:09:26.756 INFO:journalctl@ceph.mon.b.smithi086.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 371.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-04T16:09:26.756 INFO:journalctl@ceph.mon.b.smithi086.stdout: 2024-03-04T16:09:26.756 INFO:journalctl@ceph.mon.b.smithi086.stdout: ** Compaction Stats [default] ** 2024-03-04T16:09:26.756 INFO:journalctl@ceph.mon.b.smithi086.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T16:09:26.756 INFO:journalctl@ceph.mon.b.smithi086.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T16:09:26.756 INFO:journalctl@ceph.mon.b.smithi086.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 371.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-04T16:09:26.757 INFO:journalctl@ceph.mon.b.smithi086.stdout: 2024-03-04T16:09:26.757 INFO:journalctl@ceph.mon.b.smithi086.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-04T16:09:26.757 INFO:journalctl@ceph.mon.b.smithi086.stdout: 2024-03-04T16:09:26.757 INFO:journalctl@ceph.mon.b.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T16:09:26.757 INFO:journalctl@ceph.mon.b.smithi086.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-03-04T16:09:26.757 INFO:journalctl@ceph.mon.b.smithi086.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T16:09:26.757 INFO:journalctl@ceph.mon.b.smithi086.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T16:09:26.757 INFO:journalctl@ceph.mon.b.smithi086.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T16:09:26.757 INFO:journalctl@ceph.mon.b.smithi086.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T16:09:26.757 INFO:journalctl@ceph.mon.b.smithi086.stdout: Cumulative compaction: 0.00 GB write, 124.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T16:09:26.757 INFO:journalctl@ceph.mon.b.smithi086.stdout: Interval compaction: 0.00 GB write, 124.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T16:09:26.757 INFO:journalctl@ceph.mon.b.smithi086.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T16:09:26.757 INFO:journalctl@ceph.mon.b.smithi086.stdout: Block cache BinnedLRUCache@0x5640b6cf3090#2 capacity: 512.00 MB usage: 2.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-04T16:09:26.757 INFO:journalctl@ceph.mon.b.smithi086.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.28 KB,0.000244379%) IndexBlock(1,1.58 KB,0.000301003%) Misc(1,0.00 KB,0%) 2024-03-04T16:09:26.758 INFO:journalctl@ceph.mon.b.smithi086.stdout: 2024-03-04T16:09:26.758 INFO:journalctl@ceph.mon.b.smithi086.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T16:09:26.758 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: starting mon.b rank 2 at public addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] at bind addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T16:09:26.758 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: mon.b@-1(???) e3 preinit fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 2024-03-04T16:09:26.758 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: mon.b@-1(???).mds e1 new map 2024-03-04T16:09:26.758 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: mon.b@-1(???).mds e1 print_map 2024-03-04T16:09:26.758 INFO:journalctl@ceph.mon.b.smithi086.stdout: e1 2024-03-04T16:09:26.758 INFO:journalctl@ceph.mon.b.smithi086.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T16:09:26.758 INFO:journalctl@ceph.mon.b.smithi086.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-04T16:09:26.758 INFO:journalctl@ceph.mon.b.smithi086.stdout: legacy client fscid: -1 2024-03-04T16:09:26.758 INFO:journalctl@ceph.mon.b.smithi086.stdout: 2024-03-04T16:09:26.758 INFO:journalctl@ceph.mon.b.smithi086.stdout: No filesystems configured 2024-03-04T16:09:26.758 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: mon.b@-1(???).osd e106 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-04T16:09:26.758 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-04T16:09:26.759 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-04T16:09:26.759 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-04T16:09:26.759 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 ceph-mon[57919]: mon.b@-1(???).paxosservice(auth 1..25) refresh upgraded, format 0 -> 3 2024-03-04T16:09:26.759 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 bash[57900]: 71557985ed5ad283b9dfab138ceafb6c19157775a91ae47f2830131208faa637 2024-03-04T16:09:26.759 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:26 smithi086 systemd[1]: Started Ceph mon.b for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:09:28.984 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:09:28 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:09:28.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:09:33.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:32 smithi083 ceph-mon[67383]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:33.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:32 smithi083 ceph-mon[64844]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:33.968 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:33.968 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:33.968 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:33.968 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:33.968 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: mon.c calling monitor election 2024-03-04T16:09:33.968 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: mon.a calling monitor election 2024-03-04T16:09:33.969 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:33.969 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:33.969 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:33.969 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T16:09:33.969 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: monmap e3: 3 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],b=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],c=[v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T16:09:33.969 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: fsmap 2024-03-04T16:09:33.969 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: osdmap e106: 8 total, 8 up, 8 in 2024-03-04T16:09:33.969 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: mgrmap e37: y(active, since 36s), standbys: x 2024-03-04T16:09:33.970 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-04T16:09:33.970 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T16:09:33.970 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T16:09:33.970 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: mon.b (rank 2) addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] is down (out of quorum) 2024-03-04T16:09:33.970 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:33.970 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:33 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:34.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[64844]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:34.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:34.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:34.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[64844]: mon.c calling monitor election 2024-03-04T16:09:34.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[64844]: mon.a calling monitor election 2024-03-04T16:09:34.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[64844]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:34.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[64844]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:34.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[64844]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:34.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[64844]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T16:09:34.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[64844]: monmap e3: 3 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],b=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],c=[v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T16:09:34.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[64844]: fsmap 2024-03-04T16:09:34.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[64844]: osdmap e106: 8 total, 8 up, 8 in 2024-03-04T16:09:34.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[64844]: mgrmap e37: y(active, since 36s), standbys: x 2024-03-04T16:09:34.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[64844]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-04T16:09:34.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[64844]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T16:09:34.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[64844]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T16:09:34.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[64844]: mon.b (rank 2) addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] is down (out of quorum) 2024-03-04T16:09:34.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:34.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:34.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[67383]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:34.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:34.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:34.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[67383]: mon.c calling monitor election 2024-03-04T16:09:34.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[67383]: mon.a calling monitor election 2024-03-04T16:09:34.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[67383]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:34.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[67383]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:34.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[67383]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:34.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[67383]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T16:09:34.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[67383]: monmap e3: 3 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],b=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],c=[v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T16:09:34.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[67383]: fsmap 2024-03-04T16:09:34.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[67383]: osdmap e106: 8 total, 8 up, 8 in 2024-03-04T16:09:34.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[67383]: mgrmap e37: y(active, since 36s), standbys: x 2024-03-04T16:09:34.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[67383]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-04T16:09:34.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[67383]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T16:09:34.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[67383]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T16:09:34.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[67383]: mon.b (rank 2) addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] is down (out of quorum) 2024-03-04T16:09:34.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:34.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:33 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:35.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:09:35 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:09:35] "GET /metrics HTTP/1.1" 200 38183 "" "Prometheus/2.33.4" 2024-03-04T16:09:36.740 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:09:36 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:09:36.475+0000 7fe862dc1700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-04T16:09:38.984 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:09:38 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:09:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:09:39.398 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:09:39.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T16:09:39.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T16:09:39.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: mon.b (rank 2) addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] is down (out of quorum) 2024-03-04T16:09:39.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:09:39.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: mon.c calling monitor election 2024-03-04T16:09:39.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: mon.a calling monitor election 2024-03-04T16:09:39.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:09:39.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:09:39.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: mon.b calling monitor election 2024-03-04T16:09:39.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:39.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:39.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:39.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:39.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T16:09:39.400 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T16:09:39.400 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T16:09:39.400 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: mon.b (rank 2) addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] is down (out of quorum) 2024-03-04T16:09:39.400 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: mon.a calling monitor election 2024-03-04T16:09:39.400 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: mon.c calling monitor election 2024-03-04T16:09:39.400 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T16:09:39.400 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: monmap e4: 3 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],b=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],c=[v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T16:09:39.400 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: fsmap 2024-03-04T16:09:39.400 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: osdmap e106: 8 total, 8 up, 8 in 2024-03-04T16:09:39.400 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: mgrmap e37: y(active, since 42s), standbys: x 2024-03-04T16:09:39.400 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-04T16:09:39.400 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: Cluster is now healthy 2024-03-04T16:09:39.400 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: overall HEALTH_OK 2024-03-04T16:09:39.401 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:39.401 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:39.401 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T16:09:39.401 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T16:09:39.401 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: mon.b (rank 2) addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] is down (out of quorum) 2024-03-04T16:09:39.401 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:09:39.401 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: mon.c calling monitor election 2024-03-04T16:09:39.401 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: mon.a calling monitor election 2024-03-04T16:09:39.401 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:09:39.401 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:09:39.401 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: mon.b calling monitor election 2024-03-04T16:09:39.401 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:39.401 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:39.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:39.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:39.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T16:09:39.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T16:09:39.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T16:09:39.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: mon.b (rank 2) addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] is down (out of quorum) 2024-03-04T16:09:39.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: mon.a calling monitor election 2024-03-04T16:09:39.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: mon.c calling monitor election 2024-03-04T16:09:39.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T16:09:39.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: monmap e4: 3 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],b=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],c=[v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T16:09:39.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: fsmap 2024-03-04T16:09:39.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: osdmap e106: 8 total, 8 up, 8 in 2024-03-04T16:09:39.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: mgrmap e37: y(active, since 42s), standbys: x 2024-03-04T16:09:39.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-04T16:09:39.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: Cluster is now healthy 2024-03-04T16:09:39.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: overall HEALTH_OK 2024-03-04T16:09:39.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:39.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:39 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:39.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T16:09:39.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T16:09:39.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: mon.b (rank 2) addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] is down (out of quorum) 2024-03-04T16:09:39.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:09:39.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: mon.c calling monitor election 2024-03-04T16:09:39.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: mon.a calling monitor election 2024-03-04T16:09:39.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:09:39.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:09:39.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: mon.b calling monitor election 2024-03-04T16:09:39.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:39.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:39.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:39.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:39.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T16:09:39.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T16:09:39.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T16:09:39.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: mon.b (rank 2) addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] is down (out of quorum) 2024-03-04T16:09:39.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: mon.a calling monitor election 2024-03-04T16:09:39.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: mon.c calling monitor election 2024-03-04T16:09:39.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T16:09:39.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: monmap e4: 3 mons at {a=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],b=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],c=[v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T16:09:39.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: fsmap 2024-03-04T16:09:39.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: osdmap e106: 8 total, 8 up, 8 in 2024-03-04T16:09:39.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: mgrmap e37: y(active, since 42s), standbys: x 2024-03-04T16:09:39.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-04T16:09:39.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: Cluster is now healthy 2024-03-04T16:09:39.486 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: overall HEALTH_OK 2024-03-04T16:09:39.487 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:39.487 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:39 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:40.046 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:09:40.047 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (2m) 23s ago 7m 17.7M - 0.23.0 ba2b418f427c 3c6dc3779a0e 2024-03-04T16:09:40.047 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (108s) 2s ago 7m 46.6M - 8.3.5 dad864ee21e9 fb754d58c53f 2024-03-04T16:09:40.047 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 running (78s) 23s ago 6m 49.9M - 3.5 e1d6a67b021e d58e71ccf9db 2024-03-04T16:09:40.047 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443,9283,8765 running (68s) 2s ago 11m 448M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c33100cb8a8 2024-03-04T16:09:40.047 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:8443,9283,8765 running (102s) 23s ago 13m 491M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 164dd93609ee 2024-03-04T16:09:40.047 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (53s) 23s ago 13m 39.1M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c173a2442197 2024-03-04T16:09:40.047 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (13s) 2s ago 12m 16.7M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 71557985ed5a 2024-03-04T16:09:40.047 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (28s) 23s ago 12m 17.4M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e eff1982ba275 2024-03-04T16:09:40.047 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (2m) 23s ago 8m 18.5M - 1.3.1 1dbe0e931976 9bf54b460640 2024-03-04T16:09:40.047 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (2m) 2s ago 8m 16.0M - 1.3.1 1dbe0e931976 b72dd80b2588 2024-03-04T16:09:40.047 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (11m) 23s ago 11m 53.2M 2736M 17.2.0 e1d6a67b021e 56679a9f787b 2024-03-04T16:09:40.047 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (11m) 23s ago 11m 56.2M 2736M 17.2.0 e1d6a67b021e c1070c1a315a 2024-03-04T16:09:40.047 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (10m) 23s ago 10m 49.0M 2736M 17.2.0 e1d6a67b021e 2b913c19f581 2024-03-04T16:09:40.048 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (10m) 23s ago 10m 51.3M 2736M 17.2.0 e1d6a67b021e 9f9a4e2ec5da 2024-03-04T16:09:40.048 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (10m) 2s ago 10m 53.9M 2992M 17.2.0 e1d6a67b021e b3a97f83f1c7 2024-03-04T16:09:40.048 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (9m) 2s ago 9m 50.7M 2992M 17.2.0 e1d6a67b021e b82ca42b7353 2024-03-04T16:09:40.048 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (9m) 2s ago 9m 48.8M 2992M 17.2.0 e1d6a67b021e 18cedf14321d 2024-03-04T16:09:40.048 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (8m) 2s ago 8m 54.5M 2992M 17.2.0 e1d6a67b021e 390e1798c738 2024-03-04T16:09:40.048 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 running (73s) 2s ago 8m 58.2M - 2.33.4 514e6a882f6e b563a246eeac 2024-03-04T16:09:40.048 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (6m) 23s ago 6m 73.5M - 17.2.0 e1d6a67b021e 65f14042f354 2024-03-04T16:09:40.048 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (6m) 2s ago 6m 74.9M - 17.2.0 e1d6a67b021e 78175cc6dc5d 2024-03-04T16:09:40.048 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (6m) 23s ago 6m 71.7M - 17.2.0 e1d6a67b021e 01dcf858bd62 2024-03-04T16:09:40.048 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (6m) 2s ago 6m 76.7M - 17.2.0 e1d6a67b021e 8f70a5ce9f0d 2024-03-04T16:09:40.437 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:09:40.437 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:09:40.437 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 3 2024-03-04T16:09:40.437 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:09:40.437 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:09:40.438 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:09:40.438 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:09:40.438 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:09:40.438 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-04T16:09:40.438 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:09:40.438 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:09:40.438 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-04T16:09:40.438 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:09:40.438 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:09:40.438 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-04T16:09:40.438 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 5 2024-03-04T16:09:40.438 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:09:40.438 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:09:40.439 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:40 smithi083 ceph-mon[67383]: from='client.44172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:40.439 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:40 smithi083 ceph-mon[67383]: from='client.44178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:40.439 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:40 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:40.439 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:40 smithi083 ceph-mon[64844]: from='client.44172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:40.439 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:40 smithi083 ceph-mon[64844]: from='client.44178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:40.439 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:40 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:40.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:40 smithi086 ceph-mon[57919]: from='client.44172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:40.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:40 smithi086 ceph-mon[57919]: from='client.44178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:40.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:40 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:40.766 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:09:40.766 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:09:40.766 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:09:40.766 INFO:teuthology.orchestra.run.smithi083.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:09:40.766 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [ 2024-03-04T16:09:40.766 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr", 2024-03-04T16:09:40.766 INFO:teuthology.orchestra.run.smithi083.stdout: "mon" 2024-03-04T16:09:40.766 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-04T16:09:40.766 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "5/23 daemons upgraded", 2024-03-04T16:09:40.766 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "Currently upgrading mon daemons", 2024-03-04T16:09:40.767 INFO:teuthology.orchestra.run.smithi083.stdout: "is_paused": false 2024-03-04T16:09:40.767 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:09:41.158 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_OK 2024-03-04T16:09:41.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:41 smithi086 ceph-mon[57919]: from='client.44184 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:41.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:41.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:41 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/889159035' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:41.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:41 smithi086 ceph-mon[57919]: from='client.54127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:41.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:41 smithi086 ceph-mon[57919]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:41.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:41 smithi083 ceph-mon[64844]: from='client.44184 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:41.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:41.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:41 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/889159035' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:41.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:41 smithi083 ceph-mon[64844]: from='client.54127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:41.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:41 smithi083 ceph-mon[64844]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:41.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:41 smithi083 ceph-mon[67383]: from='client.44184 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:41.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:41.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:41 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/889159035' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:09:41.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:41 smithi083 ceph-mon[67383]: from='client.54127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:09:41.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:41 smithi083 ceph-mon[67383]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:42 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/211899473' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:09:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:42 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:42.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:42 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/211899473' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:09:42.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:42 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:42.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:42 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/211899473' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:09:42.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:42 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:43.441 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:43 smithi086 ceph-mon[57919]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:43.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:43 smithi083 ceph-mon[64844]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:43.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:43 smithi083 ceph-mon[67383]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:44.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:44 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:44.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:44 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:44.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:44 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:44.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:44 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:09:44.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:44 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:44.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:44 smithi086 ceph-mon[57919]: Reconfiguring mon.a (monmap changed)... 2024-03-04T16:09:44.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:44 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:09:44.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:44 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:09:44.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:44 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:44.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:44 smithi086 ceph-mon[57919]: Reconfiguring daemon mon.a on smithi083 2024-03-04T16:09:44.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:44.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:44.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:44.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:09:44.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:44.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[64844]: Reconfiguring mon.a (monmap changed)... 2024-03-04T16:09:44.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:09:44.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:09:44.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:44.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[64844]: Reconfiguring daemon mon.a on smithi083 2024-03-04T16:09:44.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:44.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:44.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:44.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:09:44.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:44.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[67383]: Reconfiguring mon.a (monmap changed)... 2024-03-04T16:09:44.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:09:44.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:09:44.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:44.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:44 smithi083 ceph-mon[67383]: Reconfiguring daemon mon.a on smithi083 2024-03-04T16:09:45.960 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:09:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:09:45] "GET /metrics HTTP/1.1" 200 38258 "" "Prometheus/2.33.4" 2024-03-04T16:09:45.960 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:45 smithi083 ceph-mon[67383]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:45.960 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:45 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:45.960 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:45 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:45.960 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:45 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:09:45.960 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:45 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:09:45.960 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:45 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:45.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:45 smithi083 ceph-mon[64844]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:45.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:45 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:45.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:45 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:45.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:45 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:09:45.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:45 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:09:45.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:45 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:45.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:45 smithi086 ceph-mon[57919]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:45.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:45 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:45.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:45 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:45.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:45 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:09:45.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:45 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:09:45.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:45 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:46.918 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:46 smithi083 ceph-mon[64844]: Reconfiguring mgr.y (monmap changed)... 2024-03-04T16:09:46.919 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:46 smithi083 ceph-mon[64844]: Reconfiguring daemon mgr.y on smithi083 2024-03-04T16:09:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:46 smithi083 ceph-mon[67383]: Reconfiguring mgr.y (monmap changed)... 2024-03-04T16:09:46.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:46 smithi083 ceph-mon[67383]: Reconfiguring daemon mgr.y on smithi083 2024-03-04T16:09:46.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:46 smithi086 ceph-mon[57919]: Reconfiguring mgr.y (monmap changed)... 2024-03-04T16:09:46.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:46 smithi086 ceph-mon[57919]: Reconfiguring daemon mgr.y on smithi083 2024-03-04T16:09:47.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:47 smithi083 ceph-mon[64844]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:47.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:47 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:47.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:47 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:47.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:47 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:09:47.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:47 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:09:47.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:47 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:47.906 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:47 smithi083 ceph-mon[67383]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:47.906 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:47 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:47.906 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:47 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:47.906 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:47 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:09:47.906 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:47 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:09:47.906 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:47 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:47.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:47 smithi086 ceph-mon[57919]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:47.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:47 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:47.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:47 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:47.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:47 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:09:47.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:47 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:09:47.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:47 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:48.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:48 smithi086 ceph-mon[57919]: Reconfiguring mon.c (monmap changed)... 2024-03-04T16:09:48.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:48 smithi086 ceph-mon[57919]: Reconfiguring daemon mon.c on smithi083 2024-03-04T16:09:48.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:48 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:48.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:48 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:48.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:48 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:48.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:48 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T16:09:48.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:48 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:48.985 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:09:48 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:09:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:09:48.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:48 smithi083 ceph-mon[67383]: Reconfiguring mon.c (monmap changed)... 2024-03-04T16:09:48.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:48 smithi083 ceph-mon[67383]: Reconfiguring daemon mon.c on smithi083 2024-03-04T16:09:48.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:48 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:48.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:48 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:48.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:48 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:48.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:48 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T16:09:48.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:48 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:48.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:48 smithi083 ceph-mon[64844]: Reconfiguring mon.c (monmap changed)... 2024-03-04T16:09:48.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:48 smithi083 ceph-mon[64844]: Reconfiguring daemon mon.c on smithi083 2024-03-04T16:09:48.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:48 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:48.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:48 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:48.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:48 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:48.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:48 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T16:09:48.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:48 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:49.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:49 smithi083 ceph-mon[64844]: Reconfiguring osd.0 (monmap changed)... 2024-03-04T16:09:49.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:49 smithi083 ceph-mon[64844]: Reconfiguring daemon osd.0 on smithi083 2024-03-04T16:09:49.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:49 smithi083 ceph-mon[64844]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:49.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:49 smithi083 ceph-mon[67383]: Reconfiguring osd.0 (monmap changed)... 2024-03-04T16:09:49.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:49 smithi083 ceph-mon[67383]: Reconfiguring daemon osd.0 on smithi083 2024-03-04T16:09:49.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:49 smithi083 ceph-mon[67383]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:49.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:49 smithi086 ceph-mon[57919]: Reconfiguring osd.0 (monmap changed)... 2024-03-04T16:09:49.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:49 smithi086 ceph-mon[57919]: Reconfiguring daemon osd.0 on smithi083 2024-03-04T16:09:49.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:49 smithi086 ceph-mon[57919]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:51.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:51 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:51.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:51 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:51.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:51 smithi086 ceph-mon[57919]: Reconfiguring osd.1 (monmap changed)... 2024-03-04T16:09:51.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:51 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T16:09:51.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:51 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:51.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:51 smithi086 ceph-mon[57919]: Reconfiguring daemon osd.1 on smithi083 2024-03-04T16:09:51.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:51 smithi086 ceph-mon[57919]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:51.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:51 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:51.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:51 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:51.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:51 smithi083 ceph-mon[67383]: Reconfiguring osd.1 (monmap changed)... 2024-03-04T16:09:51.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:51 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T16:09:51.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:51 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:51.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:51 smithi083 ceph-mon[67383]: Reconfiguring daemon osd.1 on smithi083 2024-03-04T16:09:51.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:51 smithi083 ceph-mon[67383]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:51.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:51 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:51.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:51 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:51.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:51 smithi083 ceph-mon[64844]: Reconfiguring osd.1 (monmap changed)... 2024-03-04T16:09:51.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:51 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T16:09:51.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:51 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:51.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:51 smithi083 ceph-mon[64844]: Reconfiguring daemon osd.1 on smithi083 2024-03-04T16:09:51.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:51 smithi083 ceph-mon[64844]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:52.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:52 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:52.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:52 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:52.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:52 smithi083 ceph-mon[64844]: Reconfiguring osd.2 (monmap changed)... 2024-03-04T16:09:52.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:52 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T16:09:52.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:52 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:52.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:52 smithi083 ceph-mon[64844]: Reconfiguring daemon osd.2 on smithi083 2024-03-04T16:09:52.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:52 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:52.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:52 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:52.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:52 smithi083 ceph-mon[67383]: Reconfiguring osd.2 (monmap changed)... 2024-03-04T16:09:52.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:52 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T16:09:52.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:52 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:52.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:52 smithi083 ceph-mon[67383]: Reconfiguring daemon osd.2 on smithi083 2024-03-04T16:09:53.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:52 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:53.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:52 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:53.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:52 smithi086 ceph-mon[57919]: Reconfiguring osd.2 (monmap changed)... 2024-03-04T16:09:53.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:52 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T16:09:53.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:52 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:53.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:52 smithi086 ceph-mon[57919]: Reconfiguring daemon osd.2 on smithi083 2024-03-04T16:09:54.121 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:53 smithi083 ceph-mon[64844]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:54.122 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:53 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:54.122 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:53 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:54.122 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:53 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T16:09:54.122 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:53 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:54.122 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:53 smithi083 ceph-mon[67383]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:54.122 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:53 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:54.122 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:53 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:54.122 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:53 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T16:09:54.122 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:53 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:53 smithi086 ceph-mon[57919]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:53 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:53 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:53 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T16:09:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:53 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:55.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:54 smithi086 ceph-mon[57919]: Reconfiguring osd.3 (monmap changed)... 2024-03-04T16:09:55.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:54 smithi086 ceph-mon[57919]: Reconfiguring daemon osd.3 on smithi083 2024-03-04T16:09:55.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:54 smithi083 ceph-mon[67383]: Reconfiguring osd.3 (monmap changed)... 2024-03-04T16:09:55.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:54 smithi083 ceph-mon[67383]: Reconfiguring daemon osd.3 on smithi083 2024-03-04T16:09:55.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:54 smithi083 ceph-mon[64844]: Reconfiguring osd.3 (monmap changed)... 2024-03-04T16:09:55.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:54 smithi083 ceph-mon[64844]: Reconfiguring daemon osd.3 on smithi083 2024-03-04T16:09:55.789 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:09:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:09:55] "GET /metrics HTTP/1.1" 200 38258 "" "Prometheus/2.33.4" 2024-03-04T16:09:56.154 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:55 smithi086 ceph-mon[57919]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:56.154 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:55 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:56.154 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:55 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:56.154 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:55 smithi086 ceph-mon[57919]: Reconfiguring rgw.foo.smithi083.szjzkw (monmap changed)... 2024-03-04T16:09:56.154 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:55 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi083.szjzkw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:09:56.154 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:55 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:56.154 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:55 smithi086 ceph-mon[57919]: Reconfiguring daemon rgw.foo.smithi083.szjzkw on smithi083 2024-03-04T16:09:56.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:55 smithi083 ceph-mon[64844]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:56.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:55 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:56.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:55 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:56.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:55 smithi083 ceph-mon[64844]: Reconfiguring rgw.foo.smithi083.szjzkw (monmap changed)... 2024-03-04T16:09:56.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:55 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi083.szjzkw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:09:56.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:55 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:56.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:55 smithi083 ceph-mon[64844]: Reconfiguring daemon rgw.foo.smithi083.szjzkw on smithi083 2024-03-04T16:09:56.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:55 smithi083 ceph-mon[67383]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:56.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:55 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:56.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:55 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:56.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:55 smithi083 ceph-mon[67383]: Reconfiguring rgw.foo.smithi083.szjzkw (monmap changed)... 2024-03-04T16:09:56.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:55 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi083.szjzkw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:09:56.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:55 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:56.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:55 smithi083 ceph-mon[67383]: Reconfiguring daemon rgw.foo.smithi083.szjzkw on smithi083 2024-03-04T16:09:57.828 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:57.828 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:57.828 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:57 smithi083 ceph-mon[67383]: Reconfiguring rgw.smpl.smithi083.hqmkpf (monmap changed)... 2024-03-04T16:09:57.828 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi083.hqmkpf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:09:57.828 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:57.828 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:57 smithi083 ceph-mon[67383]: Reconfiguring daemon rgw.smpl.smithi083.hqmkpf on smithi083 2024-03-04T16:09:57.829 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:57 smithi083 ceph-mon[67383]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:57.829 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:57.829 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:57.829 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:57.829 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:57 smithi083 ceph-mon[64844]: Reconfiguring rgw.smpl.smithi083.hqmkpf (monmap changed)... 2024-03-04T16:09:57.829 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi083.hqmkpf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:09:57.829 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:57.829 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:57 smithi083 ceph-mon[64844]: Reconfiguring daemon rgw.smpl.smithi083.hqmkpf on smithi083 2024-03-04T16:09:57.829 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:57 smithi083 ceph-mon[64844]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:57.829 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:57.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:57.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:57.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:57 smithi086 ceph-mon[57919]: Reconfiguring rgw.smpl.smithi083.hqmkpf (monmap changed)... 2024-03-04T16:09:57.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi083.hqmkpf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:09:57.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:57.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:57 smithi086 ceph-mon[57919]: Reconfiguring daemon rgw.smpl.smithi083.hqmkpf on smithi083 2024-03-04T16:09:57.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:57 smithi086 ceph-mon[57919]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:57.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:58.912 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:09:58 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:09:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:09:58.912 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:58 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:58.913 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:58 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:58.913 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:58 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:58.913 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:58 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:09:58.913 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:58 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:09:58.913 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:58 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:58.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:58 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:58.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:58 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:58.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:58 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:58.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:58 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:09:58.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:58 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:09:58.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:58 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:58.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:58 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:58.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:58 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:58.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:58 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:09:58.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:58 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T16:09:58.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:58 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T16:09:58.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:58 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:59.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:59 smithi086 ceph-mon[57919]: Reconfiguring mon.b (monmap changed)... 2024-03-04T16:09:59.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:59 smithi086 ceph-mon[57919]: Reconfiguring daemon mon.b on smithi086 2024-03-04T16:09:59.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:09:59 smithi086 ceph-mon[57919]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:59.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:59 smithi083 ceph-mon[67383]: Reconfiguring mon.b (monmap changed)... 2024-03-04T16:09:59.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:59 smithi083 ceph-mon[67383]: Reconfiguring daemon mon.b on smithi086 2024-03-04T16:09:59.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:09:59 smithi083 ceph-mon[67383]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:59.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:59 smithi083 ceph-mon[64844]: Reconfiguring mon.b (monmap changed)... 2024-03-04T16:09:59.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:59 smithi083 ceph-mon[64844]: Reconfiguring daemon mon.b on smithi086 2024-03-04T16:09:59.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:09:59 smithi083 ceph-mon[64844]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:10:00.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:00 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:00.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:00 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:00.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:00 smithi086 ceph-mon[57919]: Reconfiguring mgr.x (monmap changed)... 2024-03-04T16:10:00.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:00 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:10:00.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:00 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:10:00.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:00 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:00.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:00 smithi086 ceph-mon[57919]: Reconfiguring daemon mgr.x on smithi086 2024-03-04T16:10:00.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:00 smithi086 ceph-mon[57919]: overall HEALTH_OK 2024-03-04T16:10:01.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:00 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:01.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:00 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:01.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:00 smithi083 ceph-mon[67383]: Reconfiguring mgr.x (monmap changed)... 2024-03-04T16:10:01.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:00 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:10:01.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:00 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:10:01.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:00 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:01.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:00 smithi083 ceph-mon[67383]: Reconfiguring daemon mgr.x on smithi086 2024-03-04T16:10:01.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:00 smithi083 ceph-mon[67383]: overall HEALTH_OK 2024-03-04T16:10:01.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:00 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:01.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:00 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:01.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:00 smithi083 ceph-mon[64844]: Reconfiguring mgr.x (monmap changed)... 2024-03-04T16:10:01.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:00 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T16:10:01.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:00 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T16:10:01.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:00 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:01.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:00 smithi083 ceph-mon[64844]: Reconfiguring daemon mgr.x on smithi086 2024-03-04T16:10:01.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:00 smithi083 ceph-mon[64844]: overall HEALTH_OK 2024-03-04T16:10:02.147 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:01 smithi086 ceph-mon[57919]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:02.147 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:01 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:02.147 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:01 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:02.147 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:01 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T16:10:02.147 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:01 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:02.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:01 smithi083 ceph-mon[64844]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:02.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:01 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:02.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:01 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:02.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:01 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T16:10:02.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:01 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:02.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:01 smithi083 ceph-mon[67383]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:02.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:01 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:02.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:01 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:02.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:01 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T16:10:02.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:01 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:03.202 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:02 smithi086 ceph-mon[57919]: Reconfiguring osd.4 (monmap changed)... 2024-03-04T16:10:03.202 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:02 smithi086 ceph-mon[57919]: Reconfiguring daemon osd.4 on smithi086 2024-03-04T16:10:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:02 smithi083 ceph-mon[64844]: Reconfiguring osd.4 (monmap changed)... 2024-03-04T16:10:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:02 smithi083 ceph-mon[64844]: Reconfiguring daemon osd.4 on smithi086 2024-03-04T16:10:03.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:02 smithi083 ceph-mon[67383]: Reconfiguring osd.4 (monmap changed)... 2024-03-04T16:10:03.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:02 smithi083 ceph-mon[67383]: Reconfiguring daemon osd.4 on smithi086 2024-03-04T16:10:04.083 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:03 smithi086 ceph-mon[57919]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:10:04.083 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:03 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:04.083 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:03 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:04.083 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:03 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T16:10:04.083 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:03 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:04.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:03 smithi083 ceph-mon[64844]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:10:04.268 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:03 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:04.268 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:03 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:04.268 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:03 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T16:10:04.268 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:03 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:04.268 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:03 smithi083 ceph-mon[67383]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:10:04.268 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:03 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:04.268 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:03 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:04.268 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:03 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T16:10:04.269 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:03 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:04.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:04 smithi086 ceph-mon[57919]: Reconfiguring osd.5 (monmap changed)... 2024-03-04T16:10:04.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:04 smithi086 ceph-mon[57919]: Reconfiguring daemon osd.5 on smithi086 2024-03-04T16:10:04.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:04 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:04.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:04 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:04.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:04 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-04T16:10:04.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:04 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:05.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:04 smithi083 ceph-mon[67383]: Reconfiguring osd.5 (monmap changed)... 2024-03-04T16:10:05.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:04 smithi083 ceph-mon[67383]: Reconfiguring daemon osd.5 on smithi086 2024-03-04T16:10:05.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:04 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:05.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:04 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:05.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:04 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-04T16:10:05.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:04 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:05.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:04 smithi083 ceph-mon[64844]: Reconfiguring osd.5 (monmap changed)... 2024-03-04T16:10:05.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:04 smithi083 ceph-mon[64844]: Reconfiguring daemon osd.5 on smithi086 2024-03-04T16:10:05.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:04 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:05.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:04 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:05.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:04 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-04T16:10:05.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:04 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:05.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:05 smithi086 ceph-mon[57919]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:05.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:05 smithi086 ceph-mon[57919]: Reconfiguring osd.6 (monmap changed)... 2024-03-04T16:10:05.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:05 smithi086 ceph-mon[57919]: Reconfiguring daemon osd.6 on smithi086 2024-03-04T16:10:05.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:10:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:10:05] "GET /metrics HTTP/1.1" 200 38255 "" "Prometheus/2.33.4" 2024-03-04T16:10:05.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:05 smithi083 ceph-mon[64844]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:05.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:05 smithi083 ceph-mon[64844]: Reconfiguring osd.6 (monmap changed)... 2024-03-04T16:10:05.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:05 smithi083 ceph-mon[64844]: Reconfiguring daemon osd.6 on smithi086 2024-03-04T16:10:05.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:05 smithi083 ceph-mon[67383]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:05.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:05 smithi083 ceph-mon[67383]: Reconfiguring osd.6 (monmap changed)... 2024-03-04T16:10:05.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:05 smithi083 ceph-mon[67383]: Reconfiguring daemon osd.6 on smithi086 2024-03-04T16:10:07.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:07.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:07.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:07 smithi086 ceph-mon[57919]: Reconfiguring osd.7 (monmap changed)... 2024-03-04T16:10:07.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-04T16:10:07.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:07.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:07 smithi086 ceph-mon[57919]: Reconfiguring daemon osd.7 on smithi086 2024-03-04T16:10:07.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:07 smithi086 ceph-mon[57919]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:07.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:07.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:07.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:07 smithi083 ceph-mon[64844]: Reconfiguring osd.7 (monmap changed)... 2024-03-04T16:10:07.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-04T16:10:07.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:07.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:07 smithi083 ceph-mon[64844]: Reconfiguring daemon osd.7 on smithi086 2024-03-04T16:10:07.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:07 smithi083 ceph-mon[64844]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:07.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:07.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:07.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:07 smithi083 ceph-mon[67383]: Reconfiguring osd.7 (monmap changed)... 2024-03-04T16:10:07.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-04T16:10:07.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:07.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:07 smithi083 ceph-mon[67383]: Reconfiguring daemon osd.7 on smithi086 2024-03-04T16:10:07.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:07 smithi083 ceph-mon[67383]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:08.660 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:08 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:08.660 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:08 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:08.660 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:08 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:08.660 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:08 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.ezumfo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:10:08.660 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:08 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:08.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:08 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:08.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:08 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:08.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:08 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:08.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:08 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.ezumfo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:10:08.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:08 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:08.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:08 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:08.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:08 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:08.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:08 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:08.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:08 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.ezumfo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:10:08.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:08 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:08.984 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:10:08 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:10:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:10:09.697 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:09 smithi086 ceph-mon[57919]: Reconfiguring rgw.foo.smithi086.ezumfo (monmap changed)... 2024-03-04T16:10:09.697 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:09 smithi086 ceph-mon[57919]: Reconfiguring daemon rgw.foo.smithi086.ezumfo on smithi086 2024-03-04T16:10:09.697 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:09 smithi086 ceph-mon[57919]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:10:09.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:09 smithi083 ceph-mon[67383]: Reconfiguring rgw.foo.smithi086.ezumfo (monmap changed)... 2024-03-04T16:10:09.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:09 smithi083 ceph-mon[67383]: Reconfiguring daemon rgw.foo.smithi086.ezumfo on smithi086 2024-03-04T16:10:09.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:09 smithi083 ceph-mon[67383]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:10:09.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:09 smithi083 ceph-mon[64844]: Reconfiguring rgw.foo.smithi086.ezumfo (monmap changed)... 2024-03-04T16:10:09.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:09 smithi083 ceph-mon[64844]: Reconfiguring daemon rgw.foo.smithi086.ezumfo on smithi086 2024-03-04T16:10:09.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:09 smithi083 ceph-mon[64844]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:10:10.856 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:10 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:10.856 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:10 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:10.856 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:10 smithi086 ceph-mon[57919]: Reconfiguring rgw.smpl.smithi086.yqigbl (monmap changed)... 2024-03-04T16:10:10.856 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:10 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.yqigbl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:10:10.856 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:10 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:10.856 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:10 smithi086 ceph-mon[57919]: Reconfiguring daemon rgw.smpl.smithi086.yqigbl on smithi086 2024-03-04T16:10:10.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:10 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:10.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:10 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:10.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:10 smithi083 ceph-mon[67383]: Reconfiguring rgw.smpl.smithi086.yqigbl (monmap changed)... 2024-03-04T16:10:10.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:10 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.yqigbl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:10:10.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:10 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:10.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:10 smithi083 ceph-mon[67383]: Reconfiguring daemon rgw.smpl.smithi086.yqigbl on smithi086 2024-03-04T16:10:10.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:10 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:10.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:10 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:10.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:10 smithi083 ceph-mon[64844]: Reconfiguring rgw.smpl.smithi086.yqigbl (monmap changed)... 2024-03-04T16:10:10.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:10 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.yqigbl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:10:10.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:10 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:10.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:10 smithi083 ceph-mon[64844]: Reconfiguring daemon rgw.smpl.smithi086.yqigbl on smithi086 2024-03-04T16:10:11.546 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:10:11.889 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[64844]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:11.889 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:11.889 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:11.889 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:10:11.889 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:11.889 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:11.889 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:11.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-04T16:10:11.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-04T16:10:11.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-04T16:10:11.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-04T16:10:11.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-04T16:10:11.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-04T16:10:11.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:11.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:11.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-04T16:10:11.890 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[67383]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:11.890 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:11.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:11.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:10:11.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:11.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:11.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:11.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-04T16:10:11.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-04T16:10:11.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-04T16:10:11.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-04T16:10:11.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-04T16:10:11.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-04T16:10:11.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:11.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:11.892 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-04T16:10:11.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:11 smithi086 ceph-mon[57919]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:11.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:11.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:11.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:10:11.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:11.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:11.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:11.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-04T16:10:11.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-04T16:10:11.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-04T16:10:11.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-04T16:10:11.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-04T16:10:11.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-04T16:10:11.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:11.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:11.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-04T16:10:12.235 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:10:12.235 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (2m) 55s ago 8m 17.7M - 0.23.0 ba2b418f427c 3c6dc3779a0e 2024-03-04T16:10:12.235 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (2m) 34s ago 7m 46.6M - 8.3.5 dad864ee21e9 fb754d58c53f 2024-03-04T16:10:12.235 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 running (110s) 55s ago 6m 49.9M - 3.5 e1d6a67b021e d58e71ccf9db 2024-03-04T16:10:12.236 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443,9283,8765 running (100s) 34s ago 12m 448M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c33100cb8a8 2024-03-04T16:10:12.236 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:8443,9283,8765 running (2m) 55s ago 14m 491M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 164dd93609ee 2024-03-04T16:10:12.236 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (85s) 55s ago 14m 39.1M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c173a2442197 2024-03-04T16:10:12.236 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (46s) 34s ago 12m 16.7M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 71557985ed5a 2024-03-04T16:10:12.236 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (60s) 55s ago 12m 17.4M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e eff1982ba275 2024-03-04T16:10:12.237 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (3m) 55s ago 8m 18.5M - 1.3.1 1dbe0e931976 9bf54b460640 2024-03-04T16:10:12.237 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (2m) 34s ago 8m 16.0M - 1.3.1 1dbe0e931976 b72dd80b2588 2024-03-04T16:10:12.237 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (12m) 55s ago 12m 53.2M 2736M 17.2.0 e1d6a67b021e 56679a9f787b 2024-03-04T16:10:12.237 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (11m) 55s ago 11m 56.2M 2736M 17.2.0 e1d6a67b021e c1070c1a315a 2024-03-04T16:10:12.237 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (11m) 55s ago 11m 49.0M 2736M 17.2.0 e1d6a67b021e 2b913c19f581 2024-03-04T16:10:12.237 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (11m) 55s ago 11m 51.3M 2736M 17.2.0 e1d6a67b021e 9f9a4e2ec5da 2024-03-04T16:10:12.238 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (10m) 34s ago 10m 53.9M 2992M 17.2.0 e1d6a67b021e b3a97f83f1c7 2024-03-04T16:10:12.238 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (10m) 34s ago 10m 50.7M 2992M 17.2.0 e1d6a67b021e b82ca42b7353 2024-03-04T16:10:12.238 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (9m) 34s ago 9m 48.8M 2992M 17.2.0 e1d6a67b021e 18cedf14321d 2024-03-04T16:10:12.238 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (9m) 34s ago 9m 54.5M 2992M 17.2.0 e1d6a67b021e 390e1798c738 2024-03-04T16:10:12.238 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 running (105s) 34s ago 8m 58.2M - 2.33.4 514e6a882f6e b563a246eeac 2024-03-04T16:10:12.238 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (7m) 55s ago 7m 73.5M - 17.2.0 e1d6a67b021e 65f14042f354 2024-03-04T16:10:12.238 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (7m) 34s ago 7m 74.9M - 17.2.0 e1d6a67b021e 78175cc6dc5d 2024-03-04T16:10:12.238 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (7m) 55s ago 7m 71.7M - 17.2.0 e1d6a67b021e 01dcf858bd62 2024-03-04T16:10:12.238 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (7m) 34s ago 7m 76.7M - 17.2.0 e1d6a67b021e 8f70a5ce9f0d 2024-03-04T16:10:12.657 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:10:12.658 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:10:12.658 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 3 2024-03-04T16:10:12.658 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:10:12.658 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:10:12.658 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:10:12.658 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:10:12.659 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:10:12.659 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-04T16:10:12.659 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:10:12.659 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:10:12.659 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-04T16:10:12.659 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:10:12.659 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:10:12.660 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-04T16:10:12.660 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 5 2024-03-04T16:10:12.660 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:10:12.660 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:10:12.940 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:12 smithi083 ceph-mon[67383]: Upgrade: Setting container_image for all mon 2024-03-04T16:10:12.940 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:12 smithi083 ceph-mon[67383]: Upgrade: Setting container_image for all crash 2024-03-04T16:10:12.940 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:12 smithi083 ceph-mon[67383]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-04T16:10:12.940 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:12 smithi083 ceph-mon[67383]: Upgrade: osd.0 is safe to restart 2024-03-04T16:10:12.940 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:12 smithi083 ceph-mon[67383]: from='client.44208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:12.940 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:12 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:12.940 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:12 smithi083 ceph-mon[67383]: from='client.44214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:12.941 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:12 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/784526827' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:12.941 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:12 smithi083 ceph-mon[64844]: Upgrade: Setting container_image for all mon 2024-03-04T16:10:12.941 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:12 smithi083 ceph-mon[64844]: Upgrade: Setting container_image for all crash 2024-03-04T16:10:12.941 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:12 smithi083 ceph-mon[64844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-04T16:10:12.941 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:12 smithi083 ceph-mon[64844]: Upgrade: osd.0 is safe to restart 2024-03-04T16:10:12.941 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:12 smithi083 ceph-mon[64844]: from='client.44208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:12.941 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:12 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:12.941 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:12 smithi083 ceph-mon[64844]: from='client.44214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:12.941 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:12 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/784526827' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:12.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:12 smithi086 ceph-mon[57919]: Upgrade: Setting container_image for all mon 2024-03-04T16:10:12.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:12 smithi086 ceph-mon[57919]: Upgrade: Setting container_image for all crash 2024-03-04T16:10:12.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:12 smithi086 ceph-mon[57919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-04T16:10:12.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:12 smithi086 ceph-mon[57919]: Upgrade: osd.0 is safe to restart 2024-03-04T16:10:12.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:12 smithi086 ceph-mon[57919]: from='client.44208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:12.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:12 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:12.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:12 smithi086 ceph-mon[57919]: from='client.44214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:12.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:12 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/784526827' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:13.000 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:10:13.000 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:10:13.000 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:10:13.000 INFO:teuthology.orchestra.run.smithi083.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:10:13.000 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [ 2024-03-04T16:10:13.000 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr", 2024-03-04T16:10:13.001 INFO:teuthology.orchestra.run.smithi083.stdout: "mon" 2024-03-04T16:10:13.001 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-04T16:10:13.001 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "5/23 daemons upgraded", 2024-03-04T16:10:13.001 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "Currently upgrading osd daemons", 2024-03-04T16:10:13.001 INFO:teuthology.orchestra.run.smithi083.stdout: "is_paused": false 2024-03-04T16:10:13.001 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:10:13.425 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_OK 2024-03-04T16:10:13.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:13 smithi086 ceph-mon[57919]: from='client.44220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:13.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:13 smithi086 ceph-mon[57919]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:10:13.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:13 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:13.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:13 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T16:10:13.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:13 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:13.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:13 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/3249098429' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:10:13.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:13 smithi083 ceph-mon[67383]: from='client.44220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:13.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:13 smithi083 ceph-mon[67383]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:10:13.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:13 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:13.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:13 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T16:10:13.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:13 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:13.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:13 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/3249098429' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:10:13.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:13 smithi083 ceph-mon[64844]: from='client.44220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:13.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:13 smithi083 ceph-mon[64844]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:10:13.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:13 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:13.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:13 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T16:10:13.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:13 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:13.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:13 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/3249098429' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:10:14.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:14 smithi083 ceph-mon[67383]: from='client.54163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:14.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:14 smithi083 ceph-mon[67383]: Upgrade: Updating osd.0 2024-03-04T16:10:14.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:14 smithi083 ceph-mon[67383]: Deploying daemon osd.0 on smithi083 2024-03-04T16:10:14.937 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:14 smithi083 ceph-mon[64844]: from='client.54163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:14.937 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:14 smithi083 ceph-mon[64844]: Upgrade: Updating osd.0 2024-03-04T16:10:14.938 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:14 smithi083 ceph-mon[64844]: Deploying daemon osd.0 on smithi083 2024-03-04T16:10:14.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:14 smithi086 ceph-mon[57919]: from='client.54163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:14.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:14 smithi086 ceph-mon[57919]: Upgrade: Updating osd.0 2024-03-04T16:10:14.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:14 smithi086 ceph-mon[57919]: Deploying daemon osd.0 on smithi083 2024-03-04T16:10:15.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:15 smithi086 ceph-mon[57919]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:15.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:15 smithi083 ceph-mon[67383]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:15.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:15 smithi083 ceph-mon[64844]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:15.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:10:15 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:10:15] "GET /metrics HTTP/1.1" 200 38259 "" "Prometheus/2.33.4" 2024-03-04T16:10:16.807 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:16 smithi083 systemd[1]: Stopping Ceph osd.0 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:10:17.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:16 smithi086 ceph-mon[57919]: osd.0 marked itself down and dead 2024-03-04T16:10:17.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:16 smithi083 ceph-mon[67383]: osd.0 marked itself down and dead 2024-03-04T16:10:17.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:16 smithi083 ceph-mon[64844]: osd.0 marked itself down and dead 2024-03-04T16:10:17.238 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:16 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0[32297]: 2024-03-04T16:10:16.806+0000 7f4382b1f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:10:17.238 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:16 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0[32297]: 2024-03-04T16:10:16.806+0000 7f4382b1f700 -1 osd.0 106 *** Got signal Terminated *** 2024-03-04T16:10:17.238 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:16 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0[32297]: 2024-03-04T16:10:16.806+0000 7f4382b1f700 -1 osd.0 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:10:17.738 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:17 smithi083 podman[70274]: 2024-03-04 16:10:17.379521684 +0000 UTC m=+0.689894598 container died 56679a9f787b30b4c82152a2d41a0973e2117a99a7aa997b1daf74967879b75d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0, release=754, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, name=centos-stream, GIT_CLEAN=True, distribution-scope=public, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, version=8) 2024-03-04T16:10:18.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:17 smithi086 ceph-mon[57919]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:18.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:17 smithi086 ceph-mon[57919]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:10:18.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:17 smithi086 ceph-mon[57919]: osdmap e107: 8 total, 7 up, 8 in 2024-03-04T16:10:18.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:17 smithi086 ceph-mon[57919]: osdmap e108: 8 total, 7 up, 8 in 2024-03-04T16:10:18.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:17 smithi083 ceph-mon[64844]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:18.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:17 smithi083 ceph-mon[64844]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:10:18.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:17 smithi083 ceph-mon[64844]: osdmap e107: 8 total, 7 up, 8 in 2024-03-04T16:10:18.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:17 smithi083 ceph-mon[64844]: osdmap e108: 8 total, 7 up, 8 in 2024-03-04T16:10:18.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:17 smithi083 ceph-mon[67383]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:18.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:17 smithi083 ceph-mon[67383]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:10:18.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:17 smithi083 ceph-mon[67383]: osdmap e107: 8 total, 7 up, 8 in 2024-03-04T16:10:18.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:17 smithi083 ceph-mon[67383]: osdmap e108: 8 total, 7 up, 8 in 2024-03-04T16:10:18.239 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:17 smithi083 podman[70274]: 2024-03-04 16:10:17.782284271 +0000 UTC m=+1.092657188 container cleanup 56679a9f787b30b4c82152a2d41a0973e2117a99a7aa997b1daf74967879b75d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., version=8, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, distribution-scope=public, RELEASE=HEAD, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, name=centos-stream, maintainer=Guillaume Abrioux , ceph=True) 2024-03-04T16:10:18.239 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:17 smithi083 bash[70274]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0 2024-03-04T16:10:18.239 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:18 smithi083 podman[70291]: 2024-03-04 16:10:18.001169053 +0000 UTC m=+0.619612209 container remove 56679a9f787b30b4c82152a2d41a0973e2117a99a7aa997b1daf74967879b75d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , GIT_CLEAN=True, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, release=754, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, version=8, io.openshift.expose-services=) 2024-03-04T16:10:18.646 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:18 smithi083 podman[70380]: 2024-03-04 16:10:18.2424405 +0000 UTC m=+0.018400227 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:10:18.646 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:18 smithi083 podman[70380]: 2024-03-04 16:10:18.352021796 +0000 UTC m=+0.127981532 container create d805ac549879f3ac09f9755d2d59fd4729aa2b3ceda7ab6660998c2b8c43c75f (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-04T16:10:18.646 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:18 smithi083 podman[70380]: 2024-03-04 16:10:18.563009095 +0000 UTC m=+0.338968834 container init d805ac549879f3ac09f9755d2d59fd4729aa2b3ceda7ab6660998c2b8c43c75f (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-04T16:10:18.646 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:18 smithi083 podman[70380]: 2024-03-04 16:10:18.566057572 +0000 UTC m=+0.342017302 container start d805ac549879f3ac09f9755d2d59fd4729aa2b3ceda7ab6660998c2b8c43c75f (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-04T16:10:18.646 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:18 smithi083 podman[70380]: 2024-03-04 16:10:18.646235644 +0000 UTC m=+0.422195391 container attach d805ac549879f3ac09f9755d2d59fd4729aa2b3ceda7ab6660998c2b8c43c75f (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240222) 2024-03-04T16:10:18.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:18 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:18.984 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:10:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:10:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:10:18.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:18 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:18.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:18 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:18.988 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:18 smithi083 podman[70380]: 2024-03-04 16:10:18.857294065 +0000 UTC m=+0.633253784 container died d805ac549879f3ac09f9755d2d59fd4729aa2b3ceda7ab6660998c2b8c43c75f (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-04T16:10:19.606 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:19 smithi083 podman[70406]: 2024-03-04 16:10:19.592779864 +0000 UTC m=+0.729938000 container remove d805ac549879f3ac09f9755d2d59fd4729aa2b3ceda7ab6660998c2b8c43c75f (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-04T16:10:19.876 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:19 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-04T16:10:19.876 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:19 smithi083 systemd[1]: Stopped Ceph osd.0 for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:10:19.876 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:19 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.0.service: Consumed 7.111s CPU time. 2024-03-04T16:10:19.876 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:19 smithi083 systemd[1]: Starting Ceph osd.0 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:10:20.185 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:19 smithi083 ceph-mon[64844]: pgmap v46: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:10:20.185 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:19 smithi083 ceph-mon[64844]: Health check failed: Reduced data availability: 13 pgs inactive, 56 pgs peering (PG_AVAILABILITY) 2024-03-04T16:10:20.185 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:19 smithi083 ceph-mon[67383]: pgmap v46: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:10:20.185 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:19 smithi083 ceph-mon[67383]: Health check failed: Reduced data availability: 13 pgs inactive, 56 pgs peering (PG_AVAILABILITY) 2024-03-04T16:10:20.185 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 podman[70509]: 2024-03-04 16:10:19.939531242 +0000 UTC m=+0.019370302 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:10:20.186 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 podman[70509]: 2024-03-04 16:10:20.060532033 +0000 UTC m=+0.140371082 container create 006b9eda9a54e72d4414be7f13456e909fedb120e8a16f0b5f52b6f787d9edd9 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-04T16:10:20.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:19 smithi086 ceph-mon[57919]: pgmap v46: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:10:20.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:19 smithi086 ceph-mon[57919]: Health check failed: Reduced data availability: 13 pgs inactive, 56 pgs peering (PG_AVAILABILITY) 2024-03-04T16:10:20.488 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 podman[70509]: 2024-03-04 16:10:20.296846053 +0000 UTC m=+0.376685110 container init 006b9eda9a54e72d4414be7f13456e909fedb120e8a16f0b5f52b6f787d9edd9 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-04T16:10:20.488 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 podman[70509]: 2024-03-04 16:10:20.300168269 +0000 UTC m=+0.380007322 container start 006b9eda9a54e72d4414be7f13456e909fedb120e8a16f0b5f52b6f787d9edd9 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-04T16:10:20.488 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 podman[70509]: 2024-03-04 16:10:20.379712291 +0000 UTC m=+0.459551345 container attach 006b9eda9a54e72d4414be7f13456e909fedb120e8a16f0b5f52b6f787d9edd9 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-activate, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-04T16:10:21.151 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-activate[70524]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-04T16:10:21.151 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-activate[70524]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-04T16:10:21.151 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 bash[70509]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-04T16:10:21.151 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 bash[70509]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-04T16:10:21.151 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 bash[70509]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-04T16:10:21.151 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 bash[70509]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-04T16:10:21.151 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 bash[70509]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-04T16:10:21.151 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 bash[70509]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-04T16:10:21.151 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 bash[70509]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-04T16:10:21.151 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-activate[70524]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-04T16:10:21.151 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-activate[70524]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-04T16:10:21.152 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-activate[70524]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-04T16:10:21.152 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-activate[70524]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-04T16:10:21.152 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-activate[70524]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-04T16:10:21.152 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:20 smithi083 podman[70509]: 2024-03-04 16:10:20.767193179 +0000 UTC m=+0.847032231 container died 006b9eda9a54e72d4414be7f13456e909fedb120e8a16f0b5f52b6f787d9edd9 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:10:21.874 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:21 smithi083 podman[70660]: 2024-03-04 16:10:21.53501352 +0000 UTC m=+0.759214886 container remove 006b9eda9a54e72d4414be7f13456e909fedb120e8a16f0b5f52b6f787d9edd9 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5) 2024-03-04T16:10:21.874 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:21 smithi083 podman[70726]: 2024-03-04 16:10:21.752584443 +0000 UTC m=+0.112067666 container create f7ecdc5b152057ab77f47719d440e53b494eadbb984afa1c467cbf0021db3eda (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux ) 2024-03-04T16:10:21.874 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:21 smithi083 podman[70726]: 2024-03-04 16:10:21.658883004 +0000 UTC m=+0.018366224 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:10:22.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:21 smithi086 ceph-mon[57919]: pgmap v47: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:10:22.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:21 smithi083 ceph-mon[67383]: pgmap v47: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:10:22.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:21 smithi083 ceph-mon[64844]: pgmap v47: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:10:22.239 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:21 smithi083 podman[70726]: 2024-03-04 16:10:21.996667082 +0000 UTC m=+0.356150297 container init f7ecdc5b152057ab77f47719d440e53b494eadbb984afa1c467cbf0021db3eda (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-03-04T16:10:22.239 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:21 smithi083 podman[70726]: 2024-03-04 16:10:21.999722866 +0000 UTC m=+0.359206081 container start f7ecdc5b152057ab77f47719d440e53b494eadbb984afa1c467cbf0021db3eda (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) 2024-03-04T16:10:22.239 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:22 smithi083 ceph-osd[70746]: -- 172.21.15.83:0/3561167083 <== mon.0 v2:172.21.15.83:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cace503680 con 0x55cace4cd400 2024-03-04T16:10:22.239 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:22 smithi083 bash[70726]: f7ecdc5b152057ab77f47719d440e53b494eadbb984afa1c467cbf0021db3eda 2024-03-04T16:10:22.239 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:22 smithi083 systemd[1]: Started Ceph osd.0 for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:10:22.943 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:22 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0[70742]: 2024-03-04T16:10:22.549+0000 7f4f16fef7c0 -1 Falling back to public interface 2024-03-04T16:10:23.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:23 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:23.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:23 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:23.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:23 smithi083 ceph-mon[67383]: pgmap v48: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2024-03-04T16:10:23.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:23 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:23.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:23 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:23.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:23 smithi083 ceph-mon[64844]: pgmap v48: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2024-03-04T16:10:23.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:23 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:23.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:23 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:23.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:23 smithi086 ceph-mon[57919]: pgmap v48: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2024-03-04T16:10:24.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:24 smithi086 ceph-mon[57919]: Health check failed: Degraded data redundancy: 71/630 objects degraded (11.270%), 22 pgs degraded (PG_DEGRADED) 2024-03-04T16:10:24.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:24 smithi086 ceph-mon[57919]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 56 pgs peering) 2024-03-04T16:10:24.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:24 smithi083 ceph-mon[67383]: Health check failed: Degraded data redundancy: 71/630 objects degraded (11.270%), 22 pgs degraded (PG_DEGRADED) 2024-03-04T16:10:24.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:24 smithi083 ceph-mon[67383]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 56 pgs peering) 2024-03-04T16:10:24.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:24 smithi083 ceph-mon[64844]: Health check failed: Degraded data redundancy: 71/630 objects degraded (11.270%), 22 pgs degraded (PG_DEGRADED) 2024-03-04T16:10:24.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:24 smithi083 ceph-mon[64844]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 56 pgs peering) 2024-03-04T16:10:25.462 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:25 smithi083 ceph-mon[64844]: pgmap v49: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2024-03-04T16:10:25.462 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:25 smithi083 ceph-mon[67383]: pgmap v49: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2024-03-04T16:10:25.463 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:25 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0[70742]: 2024-03-04T16:10:25.273+0000 7f4f16fef7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-03-04T16:10:25.463 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:25 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0[70742]: 2024-03-04T16:10:25.304+0000 7f4f16fef7c0 -1 osd.0 106 log_to_monitors true 2024-03-04T16:10:25.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:25 smithi086 ceph-mon[57919]: pgmap v49: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2024-03-04T16:10:25.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:10:25 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:10:25] "GET /metrics HTTP/1.1" 200 38259 "" "Prometheus/2.33.4" 2024-03-04T16:10:26.385 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:26 smithi086 ceph-mon[57919]: from='osd.0 [v2:172.21.15.83:6802/205555124,v1:172.21.15.83:6803/205555124]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T16:10:26.385 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:26 smithi086 ceph-mon[57919]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T16:10:26.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:26 smithi083 ceph-mon[67383]: from='osd.0 [v2:172.21.15.83:6802/205555124,v1:172.21.15.83:6803/205555124]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T16:10:26.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:26 smithi083 ceph-mon[67383]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T16:10:26.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:26 smithi083 ceph-mon[64844]: from='osd.0 [v2:172.21.15.83:6802/205555124,v1:172.21.15.83:6803/205555124]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T16:10:26.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:26 smithi083 ceph-mon[64844]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T16:10:26.489 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:10:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0[70742]: 2024-03-04T16:10:26.188+0000 7f4f08447700 -1 osd.0 106 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:10:27.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:27 smithi086 ceph-mon[57919]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T16:10:27.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:27 smithi086 ceph-mon[57919]: from='osd.0 [v2:172.21.15.83:6802/205555124,v1:172.21.15.83:6803/205555124]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T16:10:27.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:27 smithi086 ceph-mon[57919]: osdmap e109: 8 total, 7 up, 8 in 2024-03-04T16:10:27.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:27 smithi086 ceph-mon[57919]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T16:10:27.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:27.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:27.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:27 smithi086 ceph-mon[57919]: pgmap v51: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s; 71/630 objects degraded (11.270%) 2024-03-04T16:10:27.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:27.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:27 smithi083 ceph-mon[67383]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T16:10:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:27 smithi083 ceph-mon[67383]: from='osd.0 [v2:172.21.15.83:6802/205555124,v1:172.21.15.83:6803/205555124]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T16:10:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:27 smithi083 ceph-mon[67383]: osdmap e109: 8 total, 7 up, 8 in 2024-03-04T16:10:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:27 smithi083 ceph-mon[67383]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T16:10:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:27 smithi083 ceph-mon[67383]: pgmap v51: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s; 71/630 objects degraded (11.270%) 2024-03-04T16:10:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:27 smithi083 ceph-mon[64844]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T16:10:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:27 smithi083 ceph-mon[64844]: from='osd.0 [v2:172.21.15.83:6802/205555124,v1:172.21.15.83:6803/205555124]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T16:10:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:27 smithi083 ceph-mon[64844]: osdmap e109: 8 total, 7 up, 8 in 2024-03-04T16:10:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:27 smithi083 ceph-mon[64844]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T16:10:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:27 smithi083 ceph-mon[64844]: pgmap v51: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s; 71/630 objects degraded (11.270%) 2024-03-04T16:10:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:28.462 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:28 smithi083 ceph-mon[64844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:10:28.463 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:28 smithi083 ceph-mon[64844]: osd.0 [v2:172.21.15.83:6802/205555124,v1:172.21.15.83:6803/205555124] boot 2024-03-04T16:10:28.463 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:28 smithi083 ceph-mon[64844]: osdmap e110: 8 total, 8 up, 8 in 2024-03-04T16:10:28.463 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:28 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:10:28.463 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:28 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:28.463 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:28 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:28.463 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:28 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:28.464 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:28 smithi083 ceph-mon[64844]: osdmap e111: 8 total, 8 up, 8 in 2024-03-04T16:10:28.464 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:28 smithi083 ceph-mon[67383]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:10:28.464 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:28 smithi083 ceph-mon[67383]: osd.0 [v2:172.21.15.83:6802/205555124,v1:172.21.15.83:6803/205555124] boot 2024-03-04T16:10:28.464 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:28 smithi083 ceph-mon[67383]: osdmap e110: 8 total, 8 up, 8 in 2024-03-04T16:10:28.464 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:28 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:10:28.464 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:28 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:28.464 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:28 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:28.464 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:28 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:28.464 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:28 smithi083 ceph-mon[67383]: osdmap e111: 8 total, 8 up, 8 in 2024-03-04T16:10:28.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:28 smithi086 ceph-mon[57919]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:10:28.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:28 smithi086 ceph-mon[57919]: osd.0 [v2:172.21.15.83:6802/205555124,v1:172.21.15.83:6803/205555124] boot 2024-03-04T16:10:28.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:28 smithi086 ceph-mon[57919]: osdmap e110: 8 total, 8 up, 8 in 2024-03-04T16:10:28.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:28 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:10:28.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:28 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:28.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:28 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:28.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:28 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:28.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:28 smithi086 ceph-mon[57919]: osdmap e111: 8 total, 8 up, 8 in 2024-03-04T16:10:28.984 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:10:28 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:10:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:10:29.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:29 smithi086 ceph-mon[57919]: pgmap v54: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 29/630 objects degraded (4.603%) 2024-03-04T16:10:29.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:29 smithi083 ceph-mon[64844]: pgmap v54: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 29/630 objects degraded (4.603%) 2024-03-04T16:10:29.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:29 smithi083 ceph-mon[67383]: pgmap v54: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 29/630 objects degraded (4.603%) 2024-03-04T16:10:30.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:30 smithi086 ceph-mon[57919]: Health check update: Degraded data redundancy: 29/630 objects degraded (4.603%), 8 pgs degraded (PG_DEGRADED) 2024-03-04T16:10:30.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:30 smithi083 ceph-mon[67383]: Health check update: Degraded data redundancy: 29/630 objects degraded (4.603%), 8 pgs degraded (PG_DEGRADED) 2024-03-04T16:10:30.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:30 smithi083 ceph-mon[64844]: Health check update: Degraded data redundancy: 29/630 objects degraded (4.603%), 8 pgs degraded (PG_DEGRADED) 2024-03-04T16:10:31.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:31 smithi086 ceph-mon[57919]: pgmap v55: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 29/630 objects degraded (4.603%) 2024-03-04T16:10:31.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:31 smithi083 ceph-mon[64844]: pgmap v55: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 29/630 objects degraded (4.603%) 2024-03-04T16:10:31.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:31 smithi083 ceph-mon[67383]: pgmap v55: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 29/630 objects degraded (4.603%) 2024-03-04T16:10:31.488 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:10:31 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:10:31.411+0000 7fe862dc1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-03-04T16:10:32.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:32 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:32.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:32 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:32.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:32 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:32.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:32 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:10:32.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:32 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:32.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:32 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:10:32.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:32 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:32.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:32 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:32.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:32 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:32.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:32 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:10:32.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:32 smithi086 ceph-mon[57919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:10:32.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:32 smithi086 ceph-mon[57919]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-03-04T16:10:32.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:32.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:32.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:32.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:10:32.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:32.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:10:32.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:32.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:32.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:32.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:10:32.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[64844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:10:32.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[64844]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-03-04T16:10:32.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:32.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:32.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:32.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:10:32.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:32.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:10:32.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:32.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:32.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:32.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:10:32.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[67383]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:10:32.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:32 smithi083 ceph-mon[67383]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-03-04T16:10:33.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:33 smithi086 ceph-mon[57919]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-04T16:10:33.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:33 smithi083 ceph-mon[67383]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-04T16:10:33.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:33 smithi083 ceph-mon[64844]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-04T16:10:34.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:34 smithi086 ceph-mon[57919]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/630 objects degraded (4.603%), 8 pgs degraded) 2024-03-04T16:10:34.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:34 smithi086 ceph-mon[57919]: Cluster is now healthy 2024-03-04T16:10:34.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:34 smithi083 ceph-mon[67383]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/630 objects degraded (4.603%), 8 pgs degraded) 2024-03-04T16:10:34.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:34 smithi083 ceph-mon[67383]: Cluster is now healthy 2024-03-04T16:10:34.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:34 smithi083 ceph-mon[64844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/630 objects degraded (4.603%), 8 pgs degraded) 2024-03-04T16:10:34.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:34 smithi083 ceph-mon[64844]: Cluster is now healthy 2024-03-04T16:10:35.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:35 smithi083 ceph-mon[64844]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:10:35.729 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:35 smithi083 ceph-mon[67383]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:10:35.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:35 smithi086 ceph-mon[57919]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:10:35.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:10:35 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:10:35] "GET /metrics HTTP/1.1" 200 38176 "" "Prometheus/2.33.4" 2024-03-04T16:10:38.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:37 smithi086 ceph-mon[57919]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 954 B/s rd, 0 op/s 2024-03-04T16:10:38.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:37 smithi083 ceph-mon[64844]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 954 B/s rd, 0 op/s 2024-03-04T16:10:38.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:37 smithi083 ceph-mon[67383]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 954 B/s rd, 0 op/s 2024-03-04T16:10:38.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:38 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:38.984 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:10:38 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:10:38.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-03-04T16:10:38.984 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:10:38 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:10:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:10:39.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:38 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:39.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:38 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:40.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:39 smithi086 ceph-mon[57919]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 0 op/s 2024-03-04T16:10:40.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:39 smithi083 ceph-mon[64844]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 0 op/s 2024-03-04T16:10:40.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:39 smithi083 ceph-mon[67383]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 0 op/s 2024-03-04T16:10:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:41 smithi086 ceph-mon[57919]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:42.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:41 smithi083 ceph-mon[64844]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:42.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:41 smithi083 ceph-mon[67383]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:42 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:42 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:42 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:43.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:42 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:43.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:42 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:43.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:42 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:43.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:42 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:43.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:42 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:43.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:42 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:43.784 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:10:44.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:43 smithi086 ceph-mon[57919]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:10:44.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:43 smithi083 ceph-mon[64844]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:10:44.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:43 smithi083 ceph-mon[67383]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:10:44.448 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:10:44.448 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (3m) 17s ago 8m 26.0M - 0.23.0 ba2b418f427c 3c6dc3779a0e 2024-03-04T16:10:44.448 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (2m) 67s ago 8m 46.6M - 8.3.5 dad864ee21e9 fb754d58c53f 2024-03-04T16:10:44.448 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 running (2m) 17s ago 7m 52.9M - 3.5 e1d6a67b021e d58e71ccf9db 2024-03-04T16:10:44.448 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443,9283,8765 running (2m) 67s ago 13m 448M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c33100cb8a8 2024-03-04T16:10:44.448 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:8443,9283,8765 running (2m) 17s ago 14m 501M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 164dd93609ee 2024-03-04T16:10:44.448 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (117s) 17s ago 14m 45.9M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c173a2442197 2024-03-04T16:10:44.448 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (78s) 67s ago 13m 16.7M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 71557985ed5a 2024-03-04T16:10:44.449 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (92s) 17s ago 13m 37.1M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e eff1982ba275 2024-03-04T16:10:44.449 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (3m) 17s ago 9m 18.0M - 1.3.1 1dbe0e931976 9bf54b460640 2024-03-04T16:10:44.449 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (3m) 67s ago 9m 16.0M - 1.3.1 1dbe0e931976 b72dd80b2588 2024-03-04T16:10:44.449 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (22s) 17s ago 12m 11.8M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e f7ecdc5b1520 2024-03-04T16:10:44.449 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (12m) 17s ago 12m 57.1M 2736M 17.2.0 e1d6a67b021e c1070c1a315a 2024-03-04T16:10:44.449 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (11m) 17s ago 11m 49.7M 2736M 17.2.0 e1d6a67b021e 2b913c19f581 2024-03-04T16:10:44.449 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (11m) 17s ago 11m 52.2M 2736M 17.2.0 e1d6a67b021e 9f9a4e2ec5da 2024-03-04T16:10:44.449 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (11m) 67s ago 11m 53.9M 2992M 17.2.0 e1d6a67b021e b3a97f83f1c7 2024-03-04T16:10:44.449 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (10m) 67s ago 10m 50.7M 2992M 17.2.0 e1d6a67b021e b82ca42b7353 2024-03-04T16:10:44.449 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (10m) 67s ago 10m 48.8M 2992M 17.2.0 e1d6a67b021e 18cedf14321d 2024-03-04T16:10:44.449 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (10m) 67s ago 10m 54.5M 2992M 17.2.0 e1d6a67b021e 390e1798c738 2024-03-04T16:10:44.449 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 running (2m) 67s ago 9m 58.2M - 2.33.4 514e6a882f6e b563a246eeac 2024-03-04T16:10:44.449 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (7m) 17s ago 7m 74.0M - 17.2.0 e1d6a67b021e 65f14042f354 2024-03-04T16:10:44.449 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (7m) 67s ago 7m 74.9M - 17.2.0 e1d6a67b021e 78175cc6dc5d 2024-03-04T16:10:44.449 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (7m) 17s ago 7m 72.2M - 17.2.0 e1d6a67b021e 01dcf858bd62 2024-03-04T16:10:44.450 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (7m) 67s ago 7m 76.7M - 17.2.0 e1d6a67b021e 8f70a5ce9f0d 2024-03-04T16:10:44.836 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:10:44.836 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:10:44.836 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 3 2024-03-04T16:10:44.837 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:10:44.837 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:10:44.837 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:10:44.837 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:10:44.837 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:10:44.837 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-04T16:10:44.837 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 1 2024-03-04T16:10:44.837 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:10:44.837 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:10:44.837 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-04T16:10:44.837 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:10:44.838 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:10:44.838 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-03-04T16:10:44.838 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 6 2024-03-04T16:10:44.838 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:10:44.838 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:10:45.162 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:10:45.162 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:10:45.162 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:10:45.162 INFO:teuthology.orchestra.run.smithi083.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:10:45.162 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [ 2024-03-04T16:10:45.162 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr", 2024-03-04T16:10:45.162 INFO:teuthology.orchestra.run.smithi083.stdout: "mon" 2024-03-04T16:10:45.163 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-04T16:10:45.163 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "6/23 daemons upgraded", 2024-03-04T16:10:45.163 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "Currently upgrading osd daemons", 2024-03-04T16:10:45.163 INFO:teuthology.orchestra.run.smithi083.stdout: "is_paused": false 2024-03-04T16:10:45.163 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:10:45.163 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:44 smithi083 ceph-mon[67383]: from='client.44250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:45.163 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:44 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/3944466790' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:45.163 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:44 smithi083 ceph-mon[64844]: from='client.44250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:45.164 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:44 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/3944466790' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:45.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:44 smithi086 ceph-mon[57919]: from='client.44250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:45.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:44 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/3944466790' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:10:45.554 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_OK 2024-03-04T16:10:45.884 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:10:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:10:45] "GET /metrics HTTP/1.1" 200 38399 "" "Prometheus/2.33.4" 2024-03-04T16:10:46.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:45 smithi086 ceph-mon[57919]: from='client.54187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:46.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:45 smithi086 ceph-mon[57919]: from='client.54193 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:46.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:45 smithi086 ceph-mon[57919]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:46.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:45 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/612445399' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:10:46.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:45 smithi083 ceph-mon[67383]: from='client.54187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:46.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:45 smithi083 ceph-mon[67383]: from='client.54193 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:46.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:45 smithi083 ceph-mon[67383]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:46.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:45 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/612445399' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:10:46.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:45 smithi083 ceph-mon[64844]: from='client.54187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:46.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:45 smithi083 ceph-mon[64844]: from='client.54193 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:46.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:45 smithi083 ceph-mon[64844]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:46.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:45 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/612445399' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:10:47.145 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:46 smithi083 ceph-mon[67383]: from='client.54205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:47.146 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:46 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:10:47.146 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:46 smithi083 ceph-mon[64844]: from='client.54205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:47.146 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:46 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:10:47.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:46 smithi086 ceph-mon[57919]: from='client.54205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:10:47.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:46 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:10:48.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:47 smithi086 ceph-mon[57919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:10:48.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:47 smithi086 ceph-mon[57919]: Upgrade: osd.1 is safe to restart 2024-03-04T16:10:48.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:47 smithi086 ceph-mon[57919]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:48.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:47 smithi083 ceph-mon[64844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:10:48.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:47 smithi083 ceph-mon[64844]: Upgrade: osd.1 is safe to restart 2024-03-04T16:10:48.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:47 smithi083 ceph-mon[64844]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:48.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:47 smithi083 ceph-mon[67383]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-04T16:10:48.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:47 smithi083 ceph-mon[67383]: Upgrade: osd.1 is safe to restart 2024-03-04T16:10:48.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:47 smithi083 ceph-mon[67383]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:48.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:48 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:48.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:48 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:48.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:48 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T16:10:48.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:48 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:48.984 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:10:48 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:10:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:10:49.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:48 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:49.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:48 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:49.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:48 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T16:10:49.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:48 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:49.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:48 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:49.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:48 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:49.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:48 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T16:10:49.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:48 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:50.194 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:49 smithi083 ceph-mon[67383]: Upgrade: Updating osd.1 2024-03-04T16:10:50.194 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:49 smithi083 ceph-mon[67383]: Deploying daemon osd.1 on smithi083 2024-03-04T16:10:50.195 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:49 smithi083 ceph-mon[67383]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:10:50.195 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:49 smithi083 ceph-mon[64844]: Upgrade: Updating osd.1 2024-03-04T16:10:50.195 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:49 smithi083 ceph-mon[64844]: Deploying daemon osd.1 on smithi083 2024-03-04T16:10:50.195 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:49 smithi083 ceph-mon[64844]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:10:50.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:49 smithi086 ceph-mon[57919]: Upgrade: Updating osd.1 2024-03-04T16:10:50.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:49 smithi086 ceph-mon[57919]: Deploying daemon osd.1 on smithi083 2024-03-04T16:10:50.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:49 smithi086 ceph-mon[57919]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:10:51.562 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:51 smithi083 systemd[1]: Stopping Ceph osd.1 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:10:51.897 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:51 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1[35256]: 2024-03-04T16:10:51.560+0000 7fb31a003700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:10:51.897 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:51 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1[35256]: 2024-03-04T16:10:51.560+0000 7fb31a003700 -1 osd.1 111 *** Got signal Terminated *** 2024-03-04T16:10:51.897 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:51 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1[35256]: 2024-03-04T16:10:51.560+0000 7fb31a003700 -1 osd.1 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:10:52.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:51 smithi086 ceph-mon[57919]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:52.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:51 smithi086 ceph-mon[57919]: osd.1 marked itself down and dead 2024-03-04T16:10:52.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:51 smithi083 ceph-mon[64844]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:52.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:51 smithi083 ceph-mon[64844]: osd.1 marked itself down and dead 2024-03-04T16:10:52.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:51 smithi083 ceph-mon[67383]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:10:52.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:51 smithi083 ceph-mon[67383]: osd.1 marked itself down and dead 2024-03-04T16:10:52.646 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:52 smithi083 podman[72745]: 2024-03-04 16:10:52.35770068 +0000 UTC m=+0.912960859 container died c1070c1a315a62404937dd792322115effbc7a8fba2b82d3011c79d66497872f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1, ceph=True, io.openshift.expose-services=, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, version=8, architecture=x86_64, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream) 2024-03-04T16:10:52.898 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:52 smithi083 podman[72745]: 2024-03-04 16:10:52.811639704 +0000 UTC m=+1.366899888 container cleanup c1070c1a315a62404937dd792322115effbc7a8fba2b82d3011c79d66497872f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, version=8, GIT_CLEAN=True, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-03-04T16:10:52.899 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:52 smithi083 bash[72745]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1 2024-03-04T16:10:53.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:52 smithi086 ceph-mon[57919]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:10:53.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:52 smithi086 ceph-mon[57919]: osdmap e112: 8 total, 7 up, 8 in 2024-03-04T16:10:53.238 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:53 smithi083 podman[72762]: 2024-03-04 16:10:53.088299803 +0000 UTC m=+0.733719533 container remove c1070c1a315a62404937dd792322115effbc7a8fba2b82d3011c79d66497872f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1, release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., ceph=True, maintainer=Guillaume Abrioux , version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, distribution-scope=public, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-04T16:10:53.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:52 smithi083 ceph-mon[64844]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:10:53.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:52 smithi083 ceph-mon[64844]: osdmap e112: 8 total, 7 up, 8 in 2024-03-04T16:10:53.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:52 smithi083 ceph-mon[67383]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:10:53.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:52 smithi083 ceph-mon[67383]: osdmap e112: 8 total, 7 up, 8 in 2024-03-04T16:10:53.700 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:53 smithi083 podman[72858]: 2024-03-04 16:10:53.344607399 +0000 UTC m=+0.017870635 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:10:53.700 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:53 smithi083 podman[72858]: 2024-03-04 16:10:53.473123735 +0000 UTC m=+0.146387008 container create 82746580efeaf6074d7764b7fae8bf0f96e73aac6b7aa98c1bec665c56ea9891 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-03-04T16:10:53.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:53 smithi083 ceph-mon[64844]: pgmap v67: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:10:53.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:53 smithi083 ceph-mon[64844]: Health check failed: Reduced data availability: 9 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-03-04T16:10:53.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:53 smithi083 ceph-mon[64844]: osdmap e113: 8 total, 7 up, 8 in 2024-03-04T16:10:53.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:53 smithi083 ceph-mon[67383]: pgmap v67: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:10:53.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:53 smithi083 ceph-mon[67383]: Health check failed: Reduced data availability: 9 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-03-04T16:10:53.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:53 smithi083 ceph-mon[67383]: osdmap e113: 8 total, 7 up, 8 in 2024-03-04T16:10:53.989 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:53 smithi083 podman[72858]: 2024-03-04 16:10:53.700452738 +0000 UTC m=+0.373715977 container init 82746580efeaf6074d7764b7fae8bf0f96e73aac6b7aa98c1bec665c56ea9891 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-03-04T16:10:53.989 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:53 smithi083 podman[72858]: 2024-03-04 16:10:53.703855338 +0000 UTC m=+0.377118572 container start 82746580efeaf6074d7764b7fae8bf0f96e73aac6b7aa98c1bec665c56ea9891 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-04T16:10:53.989 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:53 smithi083 podman[72858]: 2024-03-04 16:10:53.800593732 +0000 UTC m=+0.473856978 container attach 82746580efeaf6074d7764b7fae8bf0f96e73aac6b7aa98c1bec665c56ea9891 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:10:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:53 smithi086 ceph-mon[57919]: pgmap v67: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:10:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:53 smithi086 ceph-mon[57919]: Health check failed: Reduced data availability: 9 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-03-04T16:10:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:53 smithi086 ceph-mon[57919]: osdmap e113: 8 total, 7 up, 8 in 2024-03-04T16:10:54.287 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:53 smithi083 podman[72858]: 2024-03-04 16:10:53.993857213 +0000 UTC m=+0.667120450 container died 82746580efeaf6074d7764b7fae8bf0f96e73aac6b7aa98c1bec665c56ea9891 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-04T16:10:54.538 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:54 smithi083 podman[72884]: 2024-03-04 16:10:54.504458363 +0000 UTC m=+0.505571481 container remove 82746580efeaf6074d7764b7fae8bf0f96e73aac6b7aa98c1bec665c56ea9891 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-04T16:10:54.538 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:54 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-04T16:10:54.538 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:54 smithi083 systemd[1]: Stopped Ceph osd.1 for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:10:54.538 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:54 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.1.service: Consumed 7.692s CPU time. 2024-03-04T16:10:54.945 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:54 smithi083 systemd[1]: Starting Ceph osd.1 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:10:55.199 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:54 smithi083 podman[72989]: 2024-03-04 16:10:54.848873701 +0000 UTC m=+0.017777220 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:10:55.200 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:54 smithi083 podman[72989]: 2024-03-04 16:10:54.976431715 +0000 UTC m=+0.145335264 container create 516f25ce7df2ef8435ff8e9c676cfc79c4d2c81c696888acd5a669d051798941 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-activate, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:10:55.200 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:55 smithi083 podman[72989]: 2024-03-04 16:10:55.199940875 +0000 UTC m=+0.368844384 container init 516f25ce7df2ef8435ff8e9c676cfc79c4d2c81c696888acd5a669d051798941 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-activate, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD) 2024-03-04T16:10:55.488 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:55 smithi083 podman[72989]: 2024-03-04 16:10:55.204282938 +0000 UTC m=+0.373186448 container start 516f25ce7df2ef8435ff8e9c676cfc79c4d2c81c696888acd5a669d051798941 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-activate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-04T16:10:55.488 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:55 smithi083 podman[72989]: 2024-03-04 16:10:55.283330005 +0000 UTC m=+0.452233516 container attach 516f25ce7df2ef8435ff8e9c676cfc79c4d2c81c696888acd5a669d051798941 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-activate, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222) 2024-03-04T16:10:55.903 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:10:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:10:55] "GET /metrics HTTP/1.1" 200 38399 "" "Prometheus/2.33.4" 2024-03-04T16:10:55.903 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-activate[73004]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-04T16:10:55.903 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-activate[73004]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-04T16:10:55.903 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-activate[73004]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-04T16:10:55.903 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-activate[73004]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-04T16:10:55.903 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-activate[73004]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-04T16:10:55.903 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:55 smithi083 bash[72989]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-04T16:10:55.903 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:55 smithi083 bash[72989]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-04T16:10:55.904 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:55 smithi083 bash[72989]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-04T16:10:55.904 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:55 smithi083 bash[72989]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-04T16:10:55.904 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:55 smithi083 bash[72989]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-04T16:10:55.904 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:55 smithi083 bash[72989]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-04T16:10:55.904 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:55 smithi083 bash[72989]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-04T16:10:55.904 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-activate[73004]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-04T16:10:55.904 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-activate[73004]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-04T16:10:55.904 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:55 smithi083 podman[72989]: 2024-03-04 16:10:55.66600719 +0000 UTC m=+0.834910691 container died 516f25ce7df2ef8435ff8e9c676cfc79c4d2c81c696888acd5a669d051798941 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-activate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240222) 2024-03-04T16:10:56.228 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:55 smithi083 ceph-mon[64844]: pgmap v69: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:10:56.231 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:55 smithi083 ceph-mon[67383]: pgmap v69: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:10:56.233 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:55 smithi086 ceph-mon[57919]: pgmap v69: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:10:56.488 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:56 smithi083 podman[73130]: 2024-03-04 16:10:56.228747427 +0000 UTC m=+0.557859641 container remove 516f25ce7df2ef8435ff8e9c676cfc79c4d2c81c696888acd5a669d051798941 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222) 2024-03-04T16:10:56.488 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:56 smithi083 podman[73195]: 2024-03-04 16:10:56.345045823 +0000 UTC m=+0.018209489 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:10:56.488 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:56 smithi083 podman[73195]: 2024-03-04 16:10:56.471386509 +0000 UTC m=+0.144550167 container create 2c648d9a2b38ee92144deeb11a45d1b0558fb134988ef3c83391b000dc6874ff (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-03-04T16:10:56.988 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:56 smithi083 podman[73195]: 2024-03-04 16:10:56.690958604 +0000 UTC m=+0.364122271 container init 2c648d9a2b38ee92144deeb11a45d1b0558fb134988ef3c83391b000dc6874ff (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-04T16:10:56.988 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:56 smithi083 podman[73195]: 2024-03-04 16:10:56.693805535 +0000 UTC m=+0.366969191 container start 2c648d9a2b38ee92144deeb11a45d1b0558fb134988ef3c83391b000dc6874ff (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-04T16:10:56.988 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:56 smithi083 ceph-osd[73214]: -- 172.21.15.83:0/2944681831 <== mon.0 v2:172.21.15.83:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560f42c07680 con 0x560f42bd0c00 2024-03-04T16:10:56.989 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:56 smithi083 bash[73195]: 2c648d9a2b38ee92144deeb11a45d1b0558fb134988ef3c83391b000dc6874ff 2024-03-04T16:10:56.989 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:56 smithi083 systemd[1]: Started Ceph osd.1 for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:10:57.488 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:57 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1[73210]: 2024-03-04T16:10:57.233+0000 7f6bb93537c0 -1 Falling back to public interface 2024-03-04T16:10:57.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:57 smithi083 ceph-mon[64844]: pgmap v70: 161 pgs: 1 active+undersized+degraded, 56 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 6/630 objects degraded (0.952%) 2024-03-04T16:10:57.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:57.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:57.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:57.937 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:57.937 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:57 smithi083 ceph-mon[64844]: Health check failed: Degraded data redundancy: 6/630 objects degraded (0.952%), 1 pg degraded (PG_DEGRADED) 2024-03-04T16:10:57.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:57 smithi083 ceph-mon[67383]: pgmap v70: 161 pgs: 1 active+undersized+degraded, 56 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 6/630 objects degraded (0.952%) 2024-03-04T16:10:57.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:57.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:57.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:57.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:57.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:57 smithi083 ceph-mon[67383]: Health check failed: Degraded data redundancy: 6/630 objects degraded (0.952%), 1 pg degraded (PG_DEGRADED) 2024-03-04T16:10:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:57 smithi086 ceph-mon[57919]: pgmap v70: 161 pgs: 1 active+undersized+degraded, 56 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 6/630 objects degraded (0.952%) 2024-03-04T16:10:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:10:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:57 smithi086 ceph-mon[57919]: Health check failed: Degraded data redundancy: 6/630 objects degraded (0.952%), 1 pg degraded (PG_DEGRADED) 2024-03-04T16:10:58.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:58 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:58.984 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:10:58 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:10:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:10:59.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:58 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:59.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:58 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:00.225 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:59 smithi083 ceph-mon[64844]: pgmap v71: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 88/630 objects degraded (13.968%) 2024-03-04T16:11:00.225 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:10:59 smithi083 ceph-mon[64844]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 16 pgs peering) 2024-03-04T16:11:00.225 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:59 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1[73210]: 2024-03-04T16:10:59.950+0000 7f6bb93537c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-03-04T16:11:00.225 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:10:59 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1[73210]: 2024-03-04T16:10:59.981+0000 7f6bb93537c0 -1 osd.1 111 log_to_monitors true 2024-03-04T16:11:00.226 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:59 smithi083 ceph-mon[67383]: pgmap v71: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 88/630 objects degraded (13.968%) 2024-03-04T16:11:00.226 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:10:59 smithi083 ceph-mon[67383]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 16 pgs peering) 2024-03-04T16:11:00.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:59 smithi086 ceph-mon[57919]: pgmap v71: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 88/630 objects degraded (13.968%) 2024-03-04T16:11:00.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:10:59 smithi086 ceph-mon[57919]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 16 pgs peering) 2024-03-04T16:11:01.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:00 smithi086 ceph-mon[57919]: from='osd.1 [v2:172.21.15.83:6810/2349655683,v1:172.21.15.83:6811/2349655683]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T16:11:01.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:00 smithi083 ceph-mon[67383]: from='osd.1 [v2:172.21.15.83:6810/2349655683,v1:172.21.15.83:6811/2349655683]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T16:11:01.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:00 smithi083 ceph-mon[64844]: from='osd.1 [v2:172.21.15.83:6810/2349655683,v1:172.21.15.83:6811/2349655683]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T16:11:01.238 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:11:00 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1[73210]: 2024-03-04T16:11:00.903+0000 7f6baa7ab700 -1 osd.1 111 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:11:02.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:01 smithi086 ceph-mon[57919]: pgmap v72: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 88/630 objects degraded (13.968%) 2024-03-04T16:11:02.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:01 smithi086 ceph-mon[57919]: from='osd.1 [v2:172.21.15.83:6810/2349655683,v1:172.21.15.83:6811/2349655683]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T16:11:02.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:01 smithi086 ceph-mon[57919]: osdmap e114: 8 total, 7 up, 8 in 2024-03-04T16:11:02.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:01 smithi086 ceph-mon[57919]: from='osd.1 [v2:172.21.15.83:6810/2349655683,v1:172.21.15.83:6811/2349655683]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T16:11:02.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:01 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:02.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:01 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:02.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:01 smithi083 ceph-mon[67383]: pgmap v72: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 88/630 objects degraded (13.968%) 2024-03-04T16:11:02.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:01 smithi083 ceph-mon[67383]: from='osd.1 [v2:172.21.15.83:6810/2349655683,v1:172.21.15.83:6811/2349655683]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T16:11:02.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:01 smithi083 ceph-mon[67383]: osdmap e114: 8 total, 7 up, 8 in 2024-03-04T16:11:02.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:01 smithi083 ceph-mon[67383]: from='osd.1 [v2:172.21.15.83:6810/2349655683,v1:172.21.15.83:6811/2349655683]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T16:11:02.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:01 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:02.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:01 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:02.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:01 smithi083 ceph-mon[64844]: pgmap v72: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 88/630 objects degraded (13.968%) 2024-03-04T16:11:02.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:01 smithi083 ceph-mon[64844]: from='osd.1 [v2:172.21.15.83:6810/2349655683,v1:172.21.15.83:6811/2349655683]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T16:11:02.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:01 smithi083 ceph-mon[64844]: osdmap e114: 8 total, 7 up, 8 in 2024-03-04T16:11:02.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:01 smithi083 ceph-mon[64844]: from='osd.1 [v2:172.21.15.83:6810/2349655683,v1:172.21.15.83:6811/2349655683]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T16:11:02.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:01 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:02.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:01 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:03.204 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:02 smithi083 ceph-mon[67383]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:11:03.204 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:02 smithi083 ceph-mon[67383]: osd.1 [v2:172.21.15.83:6810/2349655683,v1:172.21.15.83:6811/2349655683] boot 2024-03-04T16:11:03.204 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:02 smithi083 ceph-mon[67383]: osdmap e115: 8 total, 8 up, 8 in 2024-03-04T16:11:03.204 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:02 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:11:03.205 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:02 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:03.205 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:02 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:03.205 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:02 smithi083 ceph-mon[67383]: osdmap e116: 8 total, 8 up, 8 in 2024-03-04T16:11:03.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:02 smithi083 ceph-mon[64844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:11:03.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:02 smithi083 ceph-mon[64844]: osd.1 [v2:172.21.15.83:6810/2349655683,v1:172.21.15.83:6811/2349655683] boot 2024-03-04T16:11:03.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:02 smithi083 ceph-mon[64844]: osdmap e115: 8 total, 8 up, 8 in 2024-03-04T16:11:03.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:02 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:11:03.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:02 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:03.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:02 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:03.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:02 smithi083 ceph-mon[64844]: osdmap e116: 8 total, 8 up, 8 in 2024-03-04T16:11:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:02 smithi086 ceph-mon[57919]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:11:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:02 smithi086 ceph-mon[57919]: osd.1 [v2:172.21.15.83:6810/2349655683,v1:172.21.15.83:6811/2349655683] boot 2024-03-04T16:11:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:02 smithi086 ceph-mon[57919]: osdmap e115: 8 total, 8 up, 8 in 2024-03-04T16:11:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:02 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:11:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:02 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:02 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:02 smithi086 ceph-mon[57919]: osdmap e116: 8 total, 8 up, 8 in 2024-03-04T16:11:04.208 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:03 smithi083 ceph-mon[67383]: pgmap v75: 161 pgs: 38 peering, 8 active+undersized+degraded, 17 active+undersized, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 38/630 objects degraded (6.032%) 2024-03-04T16:11:04.209 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:03 smithi083 ceph-mon[67383]: Health check update: Degraded data redundancy: 38/630 objects degraded (6.032%), 8 pgs degraded (PG_DEGRADED) 2024-03-04T16:11:04.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:03 smithi083 ceph-mon[64844]: pgmap v75: 161 pgs: 38 peering, 8 active+undersized+degraded, 17 active+undersized, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 38/630 objects degraded (6.032%) 2024-03-04T16:11:04.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:03 smithi083 ceph-mon[64844]: Health check update: Degraded data redundancy: 38/630 objects degraded (6.032%), 8 pgs degraded (PG_DEGRADED) 2024-03-04T16:11:04.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:03 smithi086 ceph-mon[57919]: pgmap v75: 161 pgs: 38 peering, 8 active+undersized+degraded, 17 active+undersized, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 38/630 objects degraded (6.032%) 2024-03-04T16:11:04.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:03 smithi086 ceph-mon[57919]: Health check update: Degraded data redundancy: 38/630 objects degraded (6.032%), 8 pgs degraded (PG_DEGRADED) 2024-03-04T16:11:05.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:11:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:11:05] "GET /metrics HTTP/1.1" 200 38466 "" "Prometheus/2.33.4" 2024-03-04T16:11:05.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:05 smithi083 ceph-mon[67383]: pgmap v77: 161 pgs: 38 peering, 8 active+undersized+degraded, 17 active+undersized, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 38/630 objects degraded (6.032%) 2024-03-04T16:11:05.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:05 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:05.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:05 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:05.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:05 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:05.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:05 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:11:05.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:05 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:05.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:05 smithi083 ceph-mon[64844]: pgmap v77: 161 pgs: 38 peering, 8 active+undersized+degraded, 17 active+undersized, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 38/630 objects degraded (6.032%) 2024-03-04T16:11:05.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:05 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:05.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:05 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:05.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:05 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:05.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:05 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:11:05.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:05 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:06.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:05 smithi086 ceph-mon[57919]: pgmap v77: 161 pgs: 38 peering, 8 active+undersized+degraded, 17 active+undersized, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 38/630 objects degraded (6.032%) 2024-03-04T16:11:06.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:05 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:06.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:05 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:06.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:05 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:06.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:05 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:11:06.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:05 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:06.488 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:11:06 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:11:05.999+0000 7fe862dc1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-04T16:11:07.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:06 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:07.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:06 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:07.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:06 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:07.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:06 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:07.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:06 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:11:07.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:06 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:07.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:06 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:07.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:06 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:07.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:06 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:07.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:06 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:11:07.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:06 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:07.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:06 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:07.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:06 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:07.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:06 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:07.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:06 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:11:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:07 smithi086 ceph-mon[57919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:11:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:07 smithi086 ceph-mon[57919]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-04T16:11:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:07 smithi086 ceph-mon[57919]: pgmap v78: 161 pgs: 38 peering, 123 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:07 smithi086 ceph-mon[57919]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/630 objects degraded (6.032%), 8 pgs degraded) 2024-03-04T16:11:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:07 smithi086 ceph-mon[57919]: Cluster is now healthy 2024-03-04T16:11:08.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:07 smithi083 ceph-mon[67383]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:11:08.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:07 smithi083 ceph-mon[67383]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-04T16:11:08.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:07 smithi083 ceph-mon[67383]: pgmap v78: 161 pgs: 38 peering, 123 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:08.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:07 smithi083 ceph-mon[67383]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/630 objects degraded (6.032%), 8 pgs degraded) 2024-03-04T16:11:08.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:07 smithi083 ceph-mon[67383]: Cluster is now healthy 2024-03-04T16:11:08.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:07 smithi083 ceph-mon[64844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:11:08.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:07 smithi083 ceph-mon[64844]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-04T16:11:08.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:07 smithi083 ceph-mon[64844]: pgmap v78: 161 pgs: 38 peering, 123 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:08.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:07 smithi083 ceph-mon[64844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/630 objects degraded (6.032%), 8 pgs degraded) 2024-03-04T16:11:08.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:07 smithi083 ceph-mon[64844]: Cluster is now healthy 2024-03-04T16:11:08.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:08 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:08.984 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:11:08 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:11:08.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-03-04T16:11:08.984 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:11:08 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:11:08.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:11:09.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:08 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:09.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:08 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:09 smithi086 ceph-mon[57919]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:11:10.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:09 smithi083 ceph-mon[64844]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:11:10.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:09 smithi083 ceph-mon[67383]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:11:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:11 smithi086 ceph-mon[57919]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:11:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:12.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:11 smithi083 ceph-mon[64844]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:11:12.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:12.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:12.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:11 smithi083 ceph-mon[67383]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:11:12.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:12.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:14.196 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:13 smithi083 ceph-mon[64844]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:11:14.196 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:13 smithi083 ceph-mon[67383]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:11:14.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:13 smithi086 ceph-mon[57919]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:11:15.909 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:11:15.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:11:15 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:11:15] "GET /metrics HTTP/1.1" 200 38491 "" "Prometheus/2.33.4" 2024-03-04T16:11:15.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:15 smithi083 ceph-mon[67383]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-04T16:11:15.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:15 smithi083 ceph-mon[64844]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-04T16:11:16.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:15 smithi086 ceph-mon[57919]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-04T16:11:16.554 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:11:16.554 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (4m) 15s ago 9m 24.9M - 0.23.0 ba2b418f427c 3c6dc3779a0e 2024-03-04T16:11:16.554 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (3m) 99s ago 8m 46.6M - 8.3.5 dad864ee21e9 fb754d58c53f 2024-03-04T16:11:16.554 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 running (2m) 15s ago 8m 53.1M - 3.5 e1d6a67b021e d58e71ccf9db 2024-03-04T16:11:16.554 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443,9283,8765 running (2m) 99s ago 13m 448M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c33100cb8a8 2024-03-04T16:11:16.555 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:8443,9283,8765 running (3m) 15s ago 15m 503M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 164dd93609ee 2024-03-04T16:11:16.555 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (2m) 15s ago 15m 47.7M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c173a2442197 2024-03-04T16:11:16.555 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (110s) 99s ago 13m 16.7M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 71557985ed5a 2024-03-04T16:11:16.555 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (2m) 15s ago 14m 39.2M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e eff1982ba275 2024-03-04T16:11:16.555 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (4m) 15s ago 9m 18.0M - 1.3.1 1dbe0e931976 9bf54b460640 2024-03-04T16:11:16.555 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (3m) 99s ago 9m 16.0M - 1.3.1 1dbe0e931976 b72dd80b2588 2024-03-04T16:11:16.555 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (54s) 15s ago 13m 42.5M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e f7ecdc5b1520 2024-03-04T16:11:16.555 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (20s) 15s ago 12m 15.0M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 2c648d9a2b38 2024-03-04T16:11:16.555 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (12m) 15s ago 12m 51.4M 2736M 17.2.0 e1d6a67b021e 2b913c19f581 2024-03-04T16:11:16.555 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (12m) 15s ago 12m 53.2M 2736M 17.2.0 e1d6a67b021e 9f9a4e2ec5da 2024-03-04T16:11:16.555 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (11m) 99s ago 11m 53.9M 2992M 17.2.0 e1d6a67b021e b3a97f83f1c7 2024-03-04T16:11:16.555 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (11m) 99s ago 11m 50.7M 2992M 17.2.0 e1d6a67b021e b82ca42b7353 2024-03-04T16:11:16.555 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (10m) 99s ago 10m 48.8M 2992M 17.2.0 e1d6a67b021e 18cedf14321d 2024-03-04T16:11:16.555 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (10m) 99s ago 10m 54.5M 2992M 17.2.0 e1d6a67b021e 390e1798c738 2024-03-04T16:11:16.556 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 running (2m) 99s ago 9m 58.2M - 2.33.4 514e6a882f6e b563a246eeac 2024-03-04T16:11:16.556 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (8m) 15s ago 8m 74.2M - 17.2.0 e1d6a67b021e 65f14042f354 2024-03-04T16:11:16.556 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (8m) 99s ago 8m 74.9M - 17.2.0 e1d6a67b021e 78175cc6dc5d 2024-03-04T16:11:16.556 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (8m) 15s ago 8m 72.3M - 17.2.0 e1d6a67b021e 01dcf858bd62 2024-03-04T16:11:16.556 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (8m) 99s ago 8m 76.7M - 17.2.0 e1d6a67b021e 8f70a5ce9f0d 2024-03-04T16:11:16.950 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:11:16.950 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:11:16.950 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 3 2024-03-04T16:11:16.950 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:11:16.950 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:11:16.951 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:11:16.951 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:11:16.951 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:11:16.951 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-04T16:11:16.951 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:11:16.951 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:11:16.951 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:11:16.951 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-04T16:11:16.951 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:11:16.951 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:11:16.951 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-04T16:11:16.951 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 7 2024-03-04T16:11:16.951 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:11:16.951 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:11:17.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:16 smithi086 ceph-mon[57919]: from='client.44292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:17.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:16 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:17.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:16 smithi083 ceph-mon[67383]: from='client.44292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:17.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:16 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:17.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:16 smithi083 ceph-mon[64844]: from='client.44292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:17.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:16 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:17.280 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:11:17.280 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:11:17.280 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:11:17.280 INFO:teuthology.orchestra.run.smithi083.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:11:17.280 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [ 2024-03-04T16:11:17.280 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr", 2024-03-04T16:11:17.280 INFO:teuthology.orchestra.run.smithi083.stdout: "mon" 2024-03-04T16:11:17.280 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-04T16:11:17.280 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "7/23 daemons upgraded", 2024-03-04T16:11:17.280 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "Currently upgrading osd daemons", 2024-03-04T16:11:17.280 INFO:teuthology.orchestra.run.smithi083.stdout: "is_paused": false 2024-03-04T16:11:17.280 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:11:17.663 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_OK 2024-03-04T16:11:18.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:17 smithi086 ceph-mon[57919]: from='client.54229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:18.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:17 smithi086 ceph-mon[57919]: from='client.44304 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:18.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:17 smithi086 ceph-mon[57919]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:18.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:17 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/3981491254' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:18.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:17 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/565087020' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:11:18.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:17 smithi083 ceph-mon[64844]: from='client.54229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:18.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:17 smithi083 ceph-mon[64844]: from='client.44304 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:18.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:17 smithi083 ceph-mon[64844]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:18.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:17 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/3981491254' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:18.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:17 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/565087020' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:11:18.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:17 smithi083 ceph-mon[67383]: from='client.54229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:18.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:17 smithi083 ceph-mon[67383]: from='client.44304 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:18.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:17 smithi083 ceph-mon[67383]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:18.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:17 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/3981491254' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:18.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:17 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/565087020' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:11:18.928 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:11:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:11:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:11:19.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:18 smithi086 ceph-mon[57919]: from='client.54247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:19.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:18 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:19.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:18 smithi083 ceph-mon[67383]: from='client.54247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:19.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:18 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:19.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:18 smithi083 ceph-mon[64844]: from='client.54247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:19.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:18 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:20.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:19 smithi086 ceph-mon[57919]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:11:20.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:19 smithi083 ceph-mon[67383]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:11:20.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:19 smithi083 ceph-mon[64844]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:11:22.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:21 smithi086 ceph-mon[57919]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:22.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:21 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:11:22.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:21 smithi083 ceph-mon[67383]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:22.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:21 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:11:22.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:21 smithi083 ceph-mon[64844]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:22.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:21 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:11:23.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:22 smithi086 ceph-mon[57919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:11:23.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:22 smithi086 ceph-mon[57919]: Upgrade: osd.2 is safe to restart 2024-03-04T16:11:23.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:22 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:23.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:22 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T16:11:23.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:22 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:23.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:22 smithi083 ceph-mon[67383]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:11:23.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:22 smithi083 ceph-mon[67383]: Upgrade: osd.2 is safe to restart 2024-03-04T16:11:23.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:22 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:23.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:22 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T16:11:23.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:22 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:23.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:22 smithi083 ceph-mon[64844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-04T16:11:23.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:22 smithi083 ceph-mon[64844]: Upgrade: osd.2 is safe to restart 2024-03-04T16:11:23.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:22 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:23.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:22 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T16:11:23.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:22 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:24.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:23 smithi086 ceph-mon[57919]: Upgrade: Updating osd.2 2024-03-04T16:11:24.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:23 smithi086 ceph-mon[57919]: Deploying daemon osd.2 on smithi083 2024-03-04T16:11:24.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:23 smithi086 ceph-mon[57919]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:11:24.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:23 smithi083 ceph-mon[67383]: Upgrade: Updating osd.2 2024-03-04T16:11:24.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:23 smithi083 ceph-mon[67383]: Deploying daemon osd.2 on smithi083 2024-03-04T16:11:24.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:23 smithi083 ceph-mon[67383]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:11:24.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:23 smithi083 ceph-mon[64844]: Upgrade: Updating osd.2 2024-03-04T16:11:24.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:23 smithi083 ceph-mon[64844]: Deploying daemon osd.2 on smithi083 2024-03-04T16:11:24.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:23 smithi083 ceph-mon[64844]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:11:25.937 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:11:25 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:11:25] "GET /metrics HTTP/1.1" 200 38491 "" "Prometheus/2.33.4" 2024-03-04T16:11:25.937 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:25 smithi083 ceph-mon[64844]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:26.194 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:25 smithi083 ceph-mon[67383]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:26.194 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:26 smithi083 systemd[1]: Stopping Ceph osd.2 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:11:26.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:25 smithi086 ceph-mon[57919]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:26.488 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2[38193]: 2024-03-04T16:11:26.327+0000 7f527832b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:11:26.488 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2[38193]: 2024-03-04T16:11:26.327+0000 7f527832b700 -1 osd.2 116 *** Got signal Terminated *** 2024-03-04T16:11:26.488 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:26 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2[38193]: 2024-03-04T16:11:26.327+0000 7f527832b700 -1 osd.2 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:11:27.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:26 smithi086 ceph-mon[57919]: osd.2 marked itself down and dead 2024-03-04T16:11:27.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:26 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:27.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:26 smithi083 ceph-mon[64844]: osd.2 marked itself down and dead 2024-03-04T16:11:27.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:26 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:27.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:26 smithi083 ceph-mon[67383]: osd.2 marked itself down and dead 2024-03-04T16:11:27.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:26 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:27.738 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:27 smithi083 podman[75216]: 2024-03-04 16:11:27.473696478 +0000 UTC m=+1.278756635 container died 2b913c19f5811829b2348e2e57ae03f5033c507886282f139556b89c51c26fe3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, name=centos-stream, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, distribution-scope=public, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., vcs-type=git, maintainer=Guillaume Abrioux , io.openshift.expose-services=, ceph=True, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-04T16:11:28.104 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:27 smithi083 ceph-mon[64844]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:28.104 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:27 smithi083 ceph-mon[64844]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:11:28.104 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:27 smithi083 ceph-mon[64844]: osdmap e117: 8 total, 7 up, 8 in 2024-03-04T16:11:28.105 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:27 smithi083 ceph-mon[67383]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:28.105 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:27 smithi083 ceph-mon[67383]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:11:28.105 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:27 smithi083 ceph-mon[67383]: osdmap e117: 8 total, 7 up, 8 in 2024-03-04T16:11:28.105 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:27 smithi083 podman[75216]: 2024-03-04 16:11:27.902809324 +0000 UTC m=+1.707869488 container cleanup 2b913c19f5811829b2348e2e57ae03f5033c507886282f139556b89c51c26fe3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , ceph=True, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, distribution-scope=public, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, release=754) 2024-03-04T16:11:28.105 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:27 smithi083 bash[75216]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2 2024-03-04T16:11:28.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:27 smithi086 ceph-mon[57919]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:28.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:27 smithi086 ceph-mon[57919]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:11:28.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:27 smithi086 ceph-mon[57919]: osdmap e117: 8 total, 7 up, 8 in 2024-03-04T16:11:28.488 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:28 smithi083 podman[75234]: 2024-03-04 16:11:28.104787578 +0000 UTC m=+0.624216607 container remove 2b913c19f5811829b2348e2e57ae03f5033c507886282f139556b89c51c26fe3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2, name=centos-stream, architecture=x86_64, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, version=8, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-03-04T16:11:28.858 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:28 smithi083 podman[75326]: 2024-03-04 16:11:28.427009314 +0000 UTC m=+0.017245438 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:11:28.858 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:28 smithi083 podman[75326]: 2024-03-04 16:11:28.547628124 +0000 UTC m=+0.137864244 container create dd41a058faa6c4e9d757ae673c8af94fdfa6f6301833076ad00a872b7f829a1b (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222) 2024-03-04T16:11:28.858 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:28 smithi083 podman[75326]: 2024-03-04 16:11:28.758610603 +0000 UTC m=+0.348846733 container init dd41a058faa6c4e9d757ae673c8af94fdfa6f6301833076ad00a872b7f829a1b (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-03-04T16:11:28.858 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:28 smithi083 podman[75326]: 2024-03-04 16:11:28.761711256 +0000 UTC m=+0.351947380 container start dd41a058faa6c4e9d757ae673c8af94fdfa6f6301833076ad00a872b7f829a1b (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:11:28.858 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:28 smithi083 podman[75326]: 2024-03-04 16:11:28.8585368 +0000 UTC m=+0.448772931 container attach dd41a058faa6c4e9d757ae673c8af94fdfa6f6301833076ad00a872b7f829a1b (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-04T16:11:28.952 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:11:28 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:11:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:11:29.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:28 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:29.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:28 smithi086 ceph-mon[57919]: osdmap e118: 8 total, 7 up, 8 in 2024-03-04T16:11:29.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:28 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:29.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:28 smithi083 ceph-mon[64844]: osdmap e118: 8 total, 7 up, 8 in 2024-03-04T16:11:29.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:28 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:29.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:28 smithi083 ceph-mon[67383]: osdmap e118: 8 total, 7 up, 8 in 2024-03-04T16:11:29.238 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:29 smithi083 podman[75326]: 2024-03-04 16:11:29.056642865 +0000 UTC m=+0.646878989 container died dd41a058faa6c4e9d757ae673c8af94fdfa6f6301833076ad00a872b7f829a1b (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-04T16:11:29.953 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:29 smithi083 podman[75352]: 2024-03-04 16:11:29.679282208 +0000 UTC m=+0.613429035 container remove dd41a058faa6c4e9d757ae673c8af94fdfa6f6301833076ad00a872b7f829a1b (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True) 2024-03-04T16:11:29.953 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:29 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-04T16:11:29.953 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:29 smithi083 systemd[1]: Stopped Ceph osd.2 for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:11:29.953 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:29 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.2.service: Consumed 6.394s CPU time. 2024-03-04T16:11:29.953 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:29 smithi083 systemd[1]: Starting Ceph osd.2 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:11:30.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:29 smithi086 ceph-mon[57919]: pgmap v91: 161 pgs: 34 peering, 4 stale+active+clean, 123 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-03-04T16:11:30.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:29 smithi086 ceph-mon[57919]: Health check failed: Reduced data availability: 9 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-03-04T16:11:30.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:29 smithi083 ceph-mon[64844]: pgmap v91: 161 pgs: 34 peering, 4 stale+active+clean, 123 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-03-04T16:11:30.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:29 smithi083 ceph-mon[64844]: Health check failed: Reduced data availability: 9 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-03-04T16:11:30.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:29 smithi083 ceph-mon[67383]: pgmap v91: 161 pgs: 34 peering, 4 stale+active+clean, 123 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-03-04T16:11:30.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:29 smithi083 ceph-mon[67383]: Health check failed: Reduced data availability: 9 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-03-04T16:11:30.238 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 podman[75455]: 2024-03-04 16:11:30.130890532 +0000 UTC m=+0.112723446 container create d49c9c2a11e881a2a45053c18c964dec553030b5dfbafbd4469a5533b388bda1 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-04T16:11:30.238 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 podman[75455]: 2024-03-04 16:11:30.035970732 +0000 UTC m=+0.017803623 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:11:30.738 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 podman[75455]: 2024-03-04 16:11:30.358958921 +0000 UTC m=+0.340791802 container init d49c9c2a11e881a2a45053c18c964dec553030b5dfbafbd4469a5533b388bda1 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-activate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:11:30.738 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 podman[75455]: 2024-03-04 16:11:30.362371223 +0000 UTC m=+0.344204100 container start d49c9c2a11e881a2a45053c18c964dec553030b5dfbafbd4469a5533b388bda1 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-04T16:11:30.738 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 podman[75455]: 2024-03-04 16:11:30.441579593 +0000 UTC m=+0.423412487 container attach d49c9c2a11e881a2a45053c18c964dec553030b5dfbafbd4469a5533b388bda1 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:11:31.238 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-activate[75470]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-04T16:11:31.238 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-activate[75470]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-04T16:11:31.238 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 bash[75455]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-04T16:11:31.238 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 bash[75455]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-04T16:11:31.239 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 bash[75455]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-04T16:11:31.239 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 bash[75455]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-04T16:11:31.239 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 bash[75455]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-04T16:11:31.239 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 bash[75455]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-04T16:11:31.239 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 bash[75455]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-04T16:11:31.239 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-activate[75470]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-04T16:11:31.239 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-activate[75470]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-04T16:11:31.239 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-activate[75470]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-04T16:11:31.239 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-activate[75470]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-04T16:11:31.239 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-activate[75470]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-04T16:11:31.239 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:30 smithi083 podman[75455]: 2024-03-04 16:11:30.826938695 +0000 UTC m=+0.808771561 container died d49c9c2a11e881a2a45053c18c964dec553030b5dfbafbd4469a5533b388bda1 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-activate, org.label-schema.build-date=20240222, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-04T16:11:31.882 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:31 smithi083 podman[75607]: 2024-03-04 16:11:31.54644901 +0000 UTC m=+0.714435016 container remove d49c9c2a11e881a2a45053c18c964dec553030b5dfbafbd4469a5533b388bda1 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-activate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:11:31.882 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:31 smithi083 podman[75671]: 2024-03-04 16:11:31.661367171 +0000 UTC m=+0.017858001 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:11:31.882 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:31 smithi083 podman[75671]: 2024-03-04 16:11:31.780786114 +0000 UTC m=+0.137276987 container create 3c971c1f3f0a4b75eeffa8719ef3649c2706cdea3aa1c0266c7812c0941f2c80 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-04T16:11:32.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:31 smithi086 ceph-mon[57919]: pgmap v92: 161 pgs: 34 peering, 4 stale+active+clean, 123 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:11:32.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:31 smithi083 ceph-mon[64844]: pgmap v92: 161 pgs: 34 peering, 4 stale+active+clean, 123 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:11:32.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:31 smithi083 ceph-mon[67383]: pgmap v92: 161 pgs: 34 peering, 4 stale+active+clean, 123 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:11:32.238 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:32 smithi083 podman[75671]: 2024-03-04 16:11:32.000010995 +0000 UTC m=+0.356501844 container init 3c971c1f3f0a4b75eeffa8719ef3649c2706cdea3aa1c0266c7812c0941f2c80 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-04T16:11:32.238 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:32 smithi083 podman[75671]: 2024-03-04 16:11:32.00338462 +0000 UTC m=+0.359875454 container start 3c971c1f3f0a4b75eeffa8719ef3649c2706cdea3aa1c0266c7812c0941f2c80 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-04T16:11:32.238 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:32 smithi083 ceph-osd[75689]: -- 172.21.15.83:0/3366147528 <== mon.1 v2:172.21.15.83:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a68b3d5680 con 0x55a68b39ec00 2024-03-04T16:11:32.238 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:32 smithi083 bash[75671]: 3c971c1f3f0a4b75eeffa8719ef3649c2706cdea3aa1c0266c7812c0941f2c80 2024-03-04T16:11:32.238 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:32 smithi083 systemd[1]: Started Ceph osd.2 for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:11:32.972 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:32 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2[75685]: 2024-03-04T16:11:32.546+0000 7f1ce07747c0 -1 Falling back to public interface 2024-03-04T16:11:33.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:33 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:33.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:33 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:33.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:33 smithi083 ceph-mon[64844]: pgmap v93: 161 pgs: 13 active+undersized+degraded, 35 active+undersized, 113 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-03-04T16:11:33.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:33 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:33.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:33 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:33.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:33 smithi083 ceph-mon[67383]: pgmap v93: 161 pgs: 13 active+undersized+degraded, 35 active+undersized, 113 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-03-04T16:11:33.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:33 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:33.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:33 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:33.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:33 smithi086 ceph-mon[57919]: pgmap v93: 161 pgs: 13 active+undersized+degraded, 35 active+undersized, 113 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-03-04T16:11:34.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:34 smithi083 ceph-mon[64844]: Health check failed: Degraded data redundancy: 55/630 objects degraded (8.730%), 13 pgs degraded (PG_DEGRADED) 2024-03-04T16:11:34.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:34 smithi083 ceph-mon[64844]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 10 pgs peering) 2024-03-04T16:11:34.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:34 smithi083 ceph-mon[67383]: Health check failed: Degraded data redundancy: 55/630 objects degraded (8.730%), 13 pgs degraded (PG_DEGRADED) 2024-03-04T16:11:34.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:34 smithi083 ceph-mon[67383]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 10 pgs peering) 2024-03-04T16:11:34.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:34 smithi086 ceph-mon[57919]: Health check failed: Degraded data redundancy: 55/630 objects degraded (8.730%), 13 pgs degraded (PG_DEGRADED) 2024-03-04T16:11:34.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:34 smithi086 ceph-mon[57919]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 10 pgs peering) 2024-03-04T16:11:35.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:35 smithi083 ceph-mon[67383]: pgmap v94: 161 pgs: 13 active+undersized+degraded, 35 active+undersized, 113 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-03-04T16:11:35.420 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:35 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2[75685]: 2024-03-04T16:11:35.242+0000 7f1ce07747c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-03-04T16:11:35.420 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:35 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2[75685]: 2024-03-04T16:11:35.268+0000 7f1ce07747c0 -1 osd.2 116 log_to_monitors true 2024-03-04T16:11:35.421 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:35 smithi083 ceph-mon[64844]: pgmap v94: 161 pgs: 13 active+undersized+degraded, 35 active+undersized, 113 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-03-04T16:11:35.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:35 smithi086 ceph-mon[57919]: pgmap v94: 161 pgs: 13 active+undersized+degraded, 35 active+undersized, 113 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-03-04T16:11:35.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:11:35 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:11:35] "GET /metrics HTTP/1.1" 200 38487 "" "Prometheus/2.33.4" 2024-03-04T16:11:36.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:36 smithi086 ceph-mon[57919]: from='osd.2 [v2:172.21.15.83:6818/774746560,v1:172.21.15.83:6819/774746560]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T16:11:36.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:36 smithi086 ceph-mon[57919]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T16:11:36.487 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:36 smithi083 ceph-mon[67383]: from='osd.2 [v2:172.21.15.83:6818/774746560,v1:172.21.15.83:6819/774746560]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T16:11:36.487 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:36 smithi083 ceph-mon[67383]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T16:11:36.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:36 smithi083 ceph-mon[64844]: from='osd.2 [v2:172.21.15.83:6818/774746560,v1:172.21.15.83:6819/774746560]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T16:11:36.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:36 smithi083 ceph-mon[64844]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T16:11:36.987 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:11:36 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2[75685]: 2024-03-04T16:11:36.647+0000 7f1ccdbc4700 -1 osd.2 116 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:11:37.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:37 smithi086 ceph-mon[57919]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T16:11:37.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:37 smithi086 ceph-mon[57919]: osdmap e119: 8 total, 7 up, 8 in 2024-03-04T16:11:37.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:37 smithi086 ceph-mon[57919]: from='osd.2 [v2:172.21.15.83:6818/774746560,v1:172.21.15.83:6819/774746560]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T16:11:37.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:37 smithi086 ceph-mon[57919]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T16:11:37.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:37 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:37.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:37 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:37.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:37 smithi086 ceph-mon[57919]: from='osd.2 ' entity='osd.2' 2024-03-04T16:11:37.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:37 smithi086 ceph-mon[57919]: pgmap v96: 161 pgs: 13 active+undersized+degraded, 35 active+undersized, 113 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 577 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-03-04T16:11:37.487 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:37 smithi083 ceph-mon[67383]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T16:11:37.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:37 smithi083 ceph-mon[67383]: osdmap e119: 8 total, 7 up, 8 in 2024-03-04T16:11:37.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:37 smithi083 ceph-mon[67383]: from='osd.2 [v2:172.21.15.83:6818/774746560,v1:172.21.15.83:6819/774746560]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T16:11:37.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:37 smithi083 ceph-mon[67383]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T16:11:37.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:37 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:37.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:37 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:37.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:37 smithi083 ceph-mon[67383]: from='osd.2 ' entity='osd.2' 2024-03-04T16:11:37.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:37 smithi083 ceph-mon[67383]: pgmap v96: 161 pgs: 13 active+undersized+degraded, 35 active+undersized, 113 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 577 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-03-04T16:11:37.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:37 smithi083 ceph-mon[64844]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T16:11:37.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:37 smithi083 ceph-mon[64844]: osdmap e119: 8 total, 7 up, 8 in 2024-03-04T16:11:37.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:37 smithi083 ceph-mon[64844]: from='osd.2 [v2:172.21.15.83:6818/774746560,v1:172.21.15.83:6819/774746560]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T16:11:37.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:37 smithi083 ceph-mon[64844]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T16:11:37.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:37 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:37.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:37 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:37.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:37 smithi083 ceph-mon[64844]: from='osd.2 ' entity='osd.2' 2024-03-04T16:11:37.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:37 smithi083 ceph-mon[64844]: pgmap v96: 161 pgs: 13 active+undersized+degraded, 35 active+undersized, 113 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 577 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-03-04T16:11:38.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:38 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:38.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:38 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:38.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:38 smithi083 ceph-mon[64844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:11:38.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:38 smithi083 ceph-mon[64844]: osd.2 [v2:172.21.15.83:6818/774746560,v1:172.21.15.83:6819/774746560] boot 2024-03-04T16:11:38.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:38 smithi083 ceph-mon[64844]: osdmap e120: 8 total, 8 up, 8 in 2024-03-04T16:11:38.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:38 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:11:38.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:38 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:38.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:38 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:38.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:38 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:38.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:38 smithi083 ceph-mon[67383]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:11:38.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:38 smithi083 ceph-mon[67383]: osd.2 [v2:172.21.15.83:6818/774746560,v1:172.21.15.83:6819/774746560] boot 2024-03-04T16:11:38.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:38 smithi083 ceph-mon[67383]: osdmap e120: 8 total, 8 up, 8 in 2024-03-04T16:11:38.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:38 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:11:38.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:38 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:38.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:38 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:38.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:38 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:38.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:38 smithi086 ceph-mon[57919]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:11:38.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:38 smithi086 ceph-mon[57919]: osd.2 [v2:172.21.15.83:6818/774746560,v1:172.21.15.83:6819/774746560] boot 2024-03-04T16:11:38.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:38 smithi086 ceph-mon[57919]: osdmap e120: 8 total, 8 up, 8 in 2024-03-04T16:11:38.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:38 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:11:38.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:38 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:38.985 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:11:38 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:11:38.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:11:39.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:39 smithi086 ceph-mon[57919]: osdmap e121: 8 total, 8 up, 8 in 2024-03-04T16:11:39.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:39 smithi086 ceph-mon[57919]: pgmap v99: 161 pgs: 24 peering, 4 active+undersized+degraded, 20 active+undersized, 113 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 25/630 objects degraded (3.968%) 2024-03-04T16:11:39.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:39 smithi083 ceph-mon[64844]: osdmap e121: 8 total, 8 up, 8 in 2024-03-04T16:11:39.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:39 smithi083 ceph-mon[64844]: pgmap v99: 161 pgs: 24 peering, 4 active+undersized+degraded, 20 active+undersized, 113 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 25/630 objects degraded (3.968%) 2024-03-04T16:11:39.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:39 smithi083 ceph-mon[67383]: osdmap e121: 8 total, 8 up, 8 in 2024-03-04T16:11:39.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:39 smithi083 ceph-mon[67383]: pgmap v99: 161 pgs: 24 peering, 4 active+undersized+degraded, 20 active+undersized, 113 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 25/630 objects degraded (3.968%) 2024-03-04T16:11:40.893 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:40 smithi083 ceph-mon[64844]: Health check update: Degraded data redundancy: 25/630 objects degraded (3.968%), 4 pgs degraded (PG_DEGRADED) 2024-03-04T16:11:40.893 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:40 smithi083 ceph-mon[67383]: Health check update: Degraded data redundancy: 25/630 objects degraded (3.968%), 4 pgs degraded (PG_DEGRADED) 2024-03-04T16:11:40.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:40 smithi086 ceph-mon[57919]: Health check update: Degraded data redundancy: 25/630 objects degraded (3.968%), 4 pgs degraded (PG_DEGRADED) 2024-03-04T16:11:41.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:11:41.084+0000 7fe862dc1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-03-04T16:11:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:41 smithi086 ceph-mon[57919]: pgmap v100: 161 pgs: 24 peering, 4 active+undersized+degraded, 20 active+undersized, 113 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 25/630 objects degraded (3.968%) 2024-03-04T16:11:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:11:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:42.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:42.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:42.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:42.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:42.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:11:42.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:42.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:42.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:42.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[64844]: pgmap v100: 161 pgs: 24 peering, 4 active+undersized+degraded, 20 active+undersized, 113 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 25/630 objects degraded (3.968%) 2024-03-04T16:11:42.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:42.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:42.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:42.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:11:42.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:42.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:42.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:42.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:42.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:42.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:11:42.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:42.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:42.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:42.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[67383]: pgmap v100: 161 pgs: 24 peering, 4 active+undersized+degraded, 20 active+undersized, 113 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 25/630 objects degraded (3.968%) 2024-03-04T16:11:42.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:42.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:42.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:42.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:11:42.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:42.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:42.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:42.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:42.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:42.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:11:42.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:42.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:42.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:42 smithi086 ceph-mon[57919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:11:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:42 smithi086 ceph-mon[57919]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-03-04T16:11:43.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:42 smithi083 ceph-mon[64844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:11:43.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:42 smithi083 ceph-mon[64844]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-03-04T16:11:43.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:42 smithi083 ceph-mon[67383]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:11:43.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:42 smithi083 ceph-mon[67383]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-03-04T16:11:44.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:43 smithi086 ceph-mon[57919]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-04T16:11:44.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:43 smithi086 ceph-mon[57919]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/630 objects degraded (3.968%), 4 pgs degraded) 2024-03-04T16:11:44.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:43 smithi086 ceph-mon[57919]: Cluster is now healthy 2024-03-04T16:11:44.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:43 smithi083 ceph-mon[64844]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-04T16:11:44.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:43 smithi083 ceph-mon[64844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/630 objects degraded (3.968%), 4 pgs degraded) 2024-03-04T16:11:44.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:43 smithi083 ceph-mon[64844]: Cluster is now healthy 2024-03-04T16:11:44.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:43 smithi083 ceph-mon[67383]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-04T16:11:44.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:43 smithi083 ceph-mon[67383]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/630 objects degraded (3.968%), 4 pgs degraded) 2024-03-04T16:11:44.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:43 smithi083 ceph-mon[67383]: Cluster is now healthy 2024-03-04T16:11:45.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:45 smithi083 ceph-mon[67383]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:11:45.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:45 smithi083 ceph-mon[64844]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:11:45.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:11:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:11:45] "GET /metrics HTTP/1.1" 200 38502 "" "Prometheus/2.33.4" 2024-03-04T16:11:46.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:45 smithi086 ceph-mon[57919]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:11:48.019 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:11:48.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:47 smithi086 ceph-mon[57919]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:11:48.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:47 smithi083 ceph-mon[64844]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:11:48.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:47 smithi083 ceph-mon[67383]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:11:48.664 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:11:48.664 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (4m) 12s ago 9m 24.9M - 0.23.0 ba2b418f427c 3c6dc3779a0e 2024-03-04T16:11:48.664 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (3m) 2m ago 9m 46.6M - 8.3.5 dad864ee21e9 fb754d58c53f 2024-03-04T16:11:48.665 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 running (3m) 12s ago 8m 53.5M - 3.5 e1d6a67b021e d58e71ccf9db 2024-03-04T16:11:48.665 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443,9283,8765 running (3m) 2m ago 14m 448M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c33100cb8a8 2024-03-04T16:11:48.665 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:8443,9283,8765 running (3m) 12s ago 15m 505M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 164dd93609ee 2024-03-04T16:11:48.665 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (3m) 12s ago 15m 47.9M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c173a2442197 2024-03-04T16:11:48.665 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (2m) 2m ago 14m 16.7M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 71557985ed5a 2024-03-04T16:11:48.665 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (2m) 12s ago 14m 39.0M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e eff1982ba275 2024-03-04T16:11:48.665 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (4m) 12s ago 10m 17.7M - 1.3.1 1dbe0e931976 9bf54b460640 2024-03-04T16:11:48.665 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (4m) 2m ago 10m 16.0M - 1.3.1 1dbe0e931976 b72dd80b2588 2024-03-04T16:11:48.665 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (86s) 12s ago 13m 43.0M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e f7ecdc5b1520 2024-03-04T16:11:48.665 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (52s) 12s ago 13m 46.2M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 2c648d9a2b38 2024-03-04T16:11:48.665 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (16s) 12s ago 13m 16.9M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c971c1f3f0a 2024-03-04T16:11:48.665 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (12m) 12s ago 12m 54.7M 2736M 17.2.0 e1d6a67b021e 9f9a4e2ec5da 2024-03-04T16:11:48.665 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (12m) 2m ago 12m 53.9M 2992M 17.2.0 e1d6a67b021e b3a97f83f1c7 2024-03-04T16:11:48.666 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (11m) 2m ago 11m 50.7M 2992M 17.2.0 e1d6a67b021e b82ca42b7353 2024-03-04T16:11:48.666 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (11m) 2m ago 11m 48.8M 2992M 17.2.0 e1d6a67b021e 18cedf14321d 2024-03-04T16:11:48.666 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (11m) 2m ago 11m 54.5M 2992M 17.2.0 e1d6a67b021e 390e1798c738 2024-03-04T16:11:48.666 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 running (3m) 2m ago 10m 58.2M - 2.33.4 514e6a882f6e b563a246eeac 2024-03-04T16:11:48.666 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (8m) 12s ago 8m 74.3M - 17.2.0 e1d6a67b021e 65f14042f354 2024-03-04T16:11:48.666 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (8m) 2m ago 8m 74.9M - 17.2.0 e1d6a67b021e 78175cc6dc5d 2024-03-04T16:11:48.666 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (8m) 12s ago 8m 72.5M - 17.2.0 e1d6a67b021e 01dcf858bd62 2024-03-04T16:11:48.666 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (8m) 2m ago 8m 76.7M - 17.2.0 e1d6a67b021e 8f70a5ce9f0d 2024-03-04T16:11:48.952 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:11:48 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:11:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:11:49.056 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:11:49.057 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:11:49.057 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 3 2024-03-04T16:11:49.057 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:11:49.057 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:11:49.057 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:11:49.057 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:11:49.057 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:11:49.057 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-04T16:11:49.057 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 3 2024-03-04T16:11:49.058 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:11:49.058 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:11:49.058 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-04T16:11:49.058 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:11:49.058 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:11:49.058 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-03-04T16:11:49.058 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 8 2024-03-04T16:11:49.058 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:11:49.058 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:11:49.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:48 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:49.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:48 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:49.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:48 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:49.388 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:11:49.389 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:11:49.389 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:11:49.389 INFO:teuthology.orchestra.run.smithi083.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:11:49.389 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [ 2024-03-04T16:11:49.389 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr", 2024-03-04T16:11:49.389 INFO:teuthology.orchestra.run.smithi083.stdout: "mon" 2024-03-04T16:11:49.389 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-04T16:11:49.389 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "8/23 daemons upgraded", 2024-03-04T16:11:49.389 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "Currently upgrading osd daemons", 2024-03-04T16:11:49.390 INFO:teuthology.orchestra.run.smithi083.stdout: "is_paused": false 2024-03-04T16:11:49.390 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:11:49.773 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_OK 2024-03-04T16:11:50.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:49 smithi086 ceph-mon[57919]: from='client.54265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:50.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:49 smithi086 ceph-mon[57919]: from='client.54271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:50.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:49 smithi086 ceph-mon[57919]: from='client.44346 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:50.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:49 smithi086 ceph-mon[57919]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1005 B/s rd, 0 op/s 2024-03-04T16:11:50.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:49 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/3422026282' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:50.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:49 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/2028979853' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:11:50.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:49 smithi083 ceph-mon[64844]: from='client.54265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:50.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:49 smithi083 ceph-mon[64844]: from='client.54271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:50.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:49 smithi083 ceph-mon[64844]: from='client.44346 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:50.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:49 smithi083 ceph-mon[64844]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1005 B/s rd, 0 op/s 2024-03-04T16:11:50.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:49 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/3422026282' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:50.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:49 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/2028979853' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:11:50.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:49 smithi083 ceph-mon[67383]: from='client.54265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:50.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:49 smithi083 ceph-mon[67383]: from='client.54271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:50.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:49 smithi083 ceph-mon[67383]: from='client.44346 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:50.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:49 smithi083 ceph-mon[67383]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1005 B/s rd, 0 op/s 2024-03-04T16:11:50.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:49 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/3422026282' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:11:50.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:49 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/2028979853' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:11:51.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:50 smithi086 ceph-mon[57919]: from='client.54289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:51.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:50 smithi083 ceph-mon[64844]: from='client.54289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:51.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:50 smithi083 ceph-mon[67383]: from='client.54289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:11:52.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:51 smithi086 ceph-mon[57919]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:52.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:51 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:52.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:51 smithi083 ceph-mon[64844]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:52.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:51 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:52.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:51 smithi083 ceph-mon[67383]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:52.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:51 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:54.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:53 smithi086 ceph-mon[57919]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:11:54.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:53 smithi083 ceph-mon[67383]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:11:54.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:53 smithi083 ceph-mon[64844]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:11:55.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:55 smithi083 ceph-mon[64844]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:55.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:55 smithi083 ceph-mon[67383]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:55.989 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:11:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:11:55] "GET /metrics HTTP/1.1" 200 38502 "" "Prometheus/2.33.4" 2024-03-04T16:11:56.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:55 smithi086 ceph-mon[57919]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:57.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:56 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:11:57.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:56 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:57.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:56 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:57.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:56 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:11:57.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:56 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:57.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:56 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:57.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:11:57.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:57.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:57 smithi086 ceph-mon[57919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:11:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:57 smithi086 ceph-mon[57919]: Upgrade: osd.3 is safe to restart 2024-03-04T16:11:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:57 smithi086 ceph-mon[57919]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T16:11:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:58.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:57 smithi083 ceph-mon[67383]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:11:58.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:57 smithi083 ceph-mon[67383]: Upgrade: osd.3 is safe to restart 2024-03-04T16:11:58.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:57 smithi083 ceph-mon[67383]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:58.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:58.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T16:11:58.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:58.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:57 smithi083 ceph-mon[64844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-04T16:11:58.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:57 smithi083 ceph-mon[64844]: Upgrade: osd.3 is safe to restart 2024-03-04T16:11:58.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:57 smithi083 ceph-mon[64844]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:11:58.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:11:58.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T16:11:58.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:58.983 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:11:58 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:11:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:11:59.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:58 smithi086 ceph-mon[57919]: Upgrade: Updating osd.3 2024-03-04T16:11:59.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:58 smithi086 ceph-mon[57919]: Deploying daemon osd.3 on smithi083 2024-03-04T16:11:59.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:58 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:59.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:58 smithi083 ceph-mon[64844]: Upgrade: Updating osd.3 2024-03-04T16:11:59.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:58 smithi083 ceph-mon[64844]: Deploying daemon osd.3 on smithi083 2024-03-04T16:11:59.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:58 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:59.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:58 smithi083 ceph-mon[67383]: Upgrade: Updating osd.3 2024-03-04T16:11:59.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:58 smithi083 ceph-mon[67383]: Deploying daemon osd.3 on smithi083 2024-03-04T16:11:59.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:58 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:00.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:11:59 smithi083 ceph-mon[67383]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:12:00.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:11:59 smithi083 ceph-mon[64844]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:12:00.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:11:59 smithi086 ceph-mon[57919]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:12:01.266 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:00 smithi083 ceph-mon[64844]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:12:01.267 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:00 smithi083 ceph-mon[67383]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:12:01.267 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:01 smithi083 systemd[1]: Stopping Ceph osd.3 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:12:01.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:00 smithi086 ceph-mon[57919]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:12:01.738 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:01 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3[41162]: 2024-03-04T16:12:01.367+0000 7f2bc73b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:12:01.738 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:01 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3[41162]: 2024-03-04T16:12:01.367+0000 7f2bc73b9700 -1 osd.3 121 *** Got signal Terminated *** 2024-03-04T16:12:01.738 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:01 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3[41162]: 2024-03-04T16:12:01.367+0000 7f2bc73b9700 -1 osd.3 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:12:02.350 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:01 smithi083 ceph-mon[64844]: osd.3 marked itself down and dead 2024-03-04T16:12:02.350 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:01 smithi083 ceph-mon[67383]: osd.3 marked itself down and dead 2024-03-04T16:12:02.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:01 smithi086 ceph-mon[57919]: osd.3 marked itself down and dead 2024-03-04T16:12:02.645 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:02 smithi083 podman[77711]: 2024-03-04 16:12:02.350106636 +0000 UTC m=+1.085134704 container died 9f9a4e2ec5daa62ff52f7a66cab67ceb7ee09e060349ceb3cd5c0d81d1eaac6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, vcs-type=git, name=centos-stream, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., ceph=True, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-03-04T16:12:02.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:02 smithi083 podman[77711]: 2024-03-04 16:12:02.835785996 +0000 UTC m=+1.570814105 container cleanup 9f9a4e2ec5daa62ff52f7a66cab67ceb7ee09e060349ceb3cd5c0d81d1eaac6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, version=8, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_CLEAN=True, com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=) 2024-03-04T16:12:02.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:02 smithi083 bash[77711]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3 2024-03-04T16:12:03.398 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:02 smithi083 ceph-mon[64844]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:12:03.398 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:02 smithi083 ceph-mon[64844]: osdmap e122: 8 total, 7 up, 8 in 2024-03-04T16:12:03.398 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:02 smithi083 ceph-mon[64844]: pgmap v112: 161 pgs: 32 peering, 13 stale+active+clean, 116 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:12:03.398 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:02 smithi083 ceph-mon[67383]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:12:03.399 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:02 smithi083 ceph-mon[67383]: osdmap e122: 8 total, 7 up, 8 in 2024-03-04T16:12:03.399 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:02 smithi083 ceph-mon[67383]: pgmap v112: 161 pgs: 32 peering, 13 stale+active+clean, 116 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:12:03.399 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:03 smithi083 podman[77728]: 2024-03-04 16:12:03.062358591 +0000 UTC m=+0.702545604 container remove 9f9a4e2ec5daa62ff52f7a66cab67ceb7ee09e060349ceb3cd5c0d81d1eaac6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3, name=centos-stream, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0) 2024-03-04T16:12:03.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:02 smithi086 ceph-mon[57919]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:12:03.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:02 smithi086 ceph-mon[57919]: osdmap e122: 8 total, 7 up, 8 in 2024-03-04T16:12:03.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:02 smithi086 ceph-mon[57919]: pgmap v112: 161 pgs: 32 peering, 13 stale+active+clean, 116 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:12:03.708 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:03 smithi083 podman[77821]: 2024-03-04 16:12:03.301333372 +0000 UTC m=+0.017489629 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:12:03.708 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:03 smithi083 podman[77821]: 2024-03-04 16:12:03.438295159 +0000 UTC m=+0.154451398 container create 611cb8dbc922391547e8f3e11ac7008c3252e0bb7052da7f658cd08b3d0de2ae (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-04T16:12:03.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:03 smithi083 podman[77821]: 2024-03-04 16:12:03.708537288 +0000 UTC m=+0.424693548 container init 611cb8dbc922391547e8f3e11ac7008c3252e0bb7052da7f658cd08b3d0de2ae (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=) 2024-03-04T16:12:03.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:03 smithi083 podman[77821]: 2024-03-04 16:12:03.712080258 +0000 UTC m=+0.428236498 container start 611cb8dbc922391547e8f3e11ac7008c3252e0bb7052da7f658cd08b3d0de2ae (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-04T16:12:03.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:03 smithi083 podman[77821]: 2024-03-04 16:12:03.816409404 +0000 UTC m=+0.532565660 container attach 611cb8dbc922391547e8f3e11ac7008c3252e0bb7052da7f658cd08b3d0de2ae (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:12:04.295 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:03 smithi083 ceph-mon[64844]: Health check failed: Reduced data availability: 7 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-03-04T16:12:04.295 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:03 smithi083 ceph-mon[64844]: osdmap e123: 8 total, 7 up, 8 in 2024-03-04T16:12:04.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:03 smithi083 ceph-mon[67383]: Health check failed: Reduced data availability: 7 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-03-04T16:12:04.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:03 smithi083 ceph-mon[67383]: osdmap e123: 8 total, 7 up, 8 in 2024-03-04T16:12:04.295 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:04 smithi083 podman[77821]: 2024-03-04 16:12:04.007526743 +0000 UTC m=+0.723682974 container died 611cb8dbc922391547e8f3e11ac7008c3252e0bb7052da7f658cd08b3d0de2ae (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0) 2024-03-04T16:12:04.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:03 smithi086 ceph-mon[57919]: Health check failed: Reduced data availability: 7 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-03-04T16:12:04.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:03 smithi086 ceph-mon[57919]: osdmap e123: 8 total, 7 up, 8 in 2024-03-04T16:12:04.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:04 smithi083 podman[77847]: 2024-03-04 16:12:04.578206789 +0000 UTC m=+0.565435345 container remove 611cb8dbc922391547e8f3e11ac7008c3252e0bb7052da7f658cd08b3d0de2ae (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-04T16:12:04.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:04 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-04T16:12:04.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:04 smithi083 systemd[1]: Stopped Ceph osd.3 for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:12:04.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:04 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.3.service: Consumed 8.368s CPU time. 2024-03-04T16:12:04.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:04 smithi083 systemd[1]: Starting Ceph osd.3 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:12:05.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:04 smithi083 ceph-mon[67383]: pgmap v114: 161 pgs: 32 peering, 13 stale+active+clean, 116 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:12:05.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:04 smithi083 ceph-mon[64844]: pgmap v114: 161 pgs: 32 peering, 13 stale+active+clean, 116 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:12:05.290 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 podman[77953]: 2024-03-04 16:12:04.918216295 +0000 UTC m=+0.018352401 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:12:05.291 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 podman[77953]: 2024-03-04 16:12:05.037685554 +0000 UTC m=+0.137821652 container create 8b1a08038189ddfbbb4e4a9c633d14ec86b2490a34ec15888a4e4f3d283dc295 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-activate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:12:05.291 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 podman[77953]: 2024-03-04 16:12:05.290595859 +0000 UTC m=+0.390731956 container init 8b1a08038189ddfbbb4e4a9c633d14ec86b2490a34ec15888a4e4f3d283dc295 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:12:05.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:04 smithi086 ceph-mon[57919]: pgmap v114: 161 pgs: 32 peering, 13 stale+active+clean, 116 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:12:05.728 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 podman[77953]: 2024-03-04 16:12:05.293812149 +0000 UTC m=+0.393948248 container start 8b1a08038189ddfbbb4e4a9c633d14ec86b2490a34ec15888a4e4f3d283dc295 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-activate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:12:05.728 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 podman[77953]: 2024-03-04 16:12:05.382297021 +0000 UTC m=+0.482433140 container attach 8b1a08038189ddfbbb4e4a9c633d14ec86b2490a34ec15888a4e4f3d283dc295 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:12:05.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:12:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:12:05] "GET /metrics HTTP/1.1" 200 38502 "" "Prometheus/2.33.4" 2024-03-04T16:12:05.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-activate[77968]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-04T16:12:05.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-activate[77968]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-04T16:12:05.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 bash[77953]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-04T16:12:05.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 bash[77953]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-04T16:12:05.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 bash[77953]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-04T16:12:05.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 bash[77953]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-04T16:12:05.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 bash[77953]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-04T16:12:05.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 bash[77953]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-04T16:12:05.989 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 bash[77953]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-04T16:12:05.989 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-activate[77968]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-04T16:12:05.989 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-activate[77968]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-04T16:12:05.989 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-activate[77968]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-04T16:12:05.989 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-activate[77968]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-04T16:12:05.989 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-activate[77968]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-04T16:12:05.989 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:05 smithi083 podman[77953]: 2024-03-04 16:12:05.762485509 +0000 UTC m=+0.862621613 container died 8b1a08038189ddfbbb4e4a9c633d14ec86b2490a34ec15888a4e4f3d283dc295 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-04T16:12:06.678 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:06 smithi083 podman[78095]: 2024-03-04 16:12:06.394483896 +0000 UTC m=+0.622345841 container remove 8b1a08038189ddfbbb4e4a9c633d14ec86b2490a34ec15888a4e4f3d283dc295 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-activate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-04T16:12:06.678 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:06 smithi083 podman[78158]: 2024-03-04 16:12:06.51212964 +0000 UTC m=+0.018439163 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:12:06.678 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:06 smithi083 podman[78158]: 2024-03-04 16:12:06.678728866 +0000 UTC m=+0.185038427 container create 8991d467d41c30d24f0738a597f97380e7f7674de42c05dd9fe8c0a707d1599c (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-04T16:12:06.956 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:06 smithi083 ceph-mon[64844]: Health check failed: Degraded data redundancy: 43/630 objects degraded (6.825%), 13 pgs degraded (PG_DEGRADED) 2024-03-04T16:12:06.956 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:06 smithi083 ceph-mon[67383]: Health check failed: Degraded data redundancy: 43/630 objects degraded (6.825%), 13 pgs degraded (PG_DEGRADED) 2024-03-04T16:12:07.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:06 smithi086 ceph-mon[57919]: Health check failed: Degraded data redundancy: 43/630 objects degraded (6.825%), 13 pgs degraded (PG_DEGRADED) 2024-03-04T16:12:07.237 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:06 smithi083 podman[78158]: 2024-03-04 16:12:06.956630767 +0000 UTC m=+0.462940291 container init 8991d467d41c30d24f0738a597f97380e7f7674de42c05dd9fe8c0a707d1599c (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2) 2024-03-04T16:12:07.238 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:06 smithi083 podman[78158]: 2024-03-04 16:12:06.959847601 +0000 UTC m=+0.466157120 container start 8991d467d41c30d24f0738a597f97380e7f7674de42c05dd9fe8c0a707d1599c (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True) 2024-03-04T16:12:07.238 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:06 smithi083 ceph-osd[78178]: -- 172.21.15.83:0/3398315295 <== mon.1 v2:172.21.15.83:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55aa8df2b680 con 0x55aa8def5000 2024-03-04T16:12:07.238 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:07 smithi083 bash[78158]: 8991d467d41c30d24f0738a597f97380e7f7674de42c05dd9fe8c0a707d1599c 2024-03-04T16:12:07.238 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:07 smithi083 systemd[1]: Started Ceph osd.3 for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:12:07.929 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:07 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3[78174]: 2024-03-04T16:12:07.505+0000 7f1a0cfa97c0 -1 Falling back to public interface 2024-03-04T16:12:08.201 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:07 smithi083 ceph-mon[67383]: pgmap v115: 161 pgs: 13 active+undersized+degraded, 32 peering, 21 active+undersized, 95 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 43/630 objects degraded (6.825%) 2024-03-04T16:12:08.202 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:08.202 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:08.202 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:07 smithi083 ceph-mon[64844]: pgmap v115: 161 pgs: 13 active+undersized+degraded, 32 peering, 21 active+undersized, 95 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 43/630 objects degraded (6.825%) 2024-03-04T16:12:08.202 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:08.202 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:07 smithi086 ceph-mon[57919]: pgmap v115: 161 pgs: 13 active+undersized+degraded, 32 peering, 21 active+undersized, 95 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 43/630 objects degraded (6.825%) 2024-03-04T16:12:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:08.936 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:12:08 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:12:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:12:09.221 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:08 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:09.222 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:08 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:09.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:08 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:10.230 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:09 smithi083 ceph-mon[64844]: pgmap v116: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-03-04T16:12:10.230 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:09 smithi083 ceph-mon[64844]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 13 pgs peering) 2024-03-04T16:12:10.230 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:09 smithi083 ceph-mon[67383]: pgmap v116: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-03-04T16:12:10.231 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:09 smithi083 ceph-mon[67383]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 13 pgs peering) 2024-03-04T16:12:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:09 smithi086 ceph-mon[57919]: pgmap v116: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-03-04T16:12:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:09 smithi086 ceph-mon[57919]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 13 pgs peering) 2024-03-04T16:12:10.488 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:10 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3[78174]: 2024-03-04T16:12:10.227+0000 7f1a0cfa97c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-03-04T16:12:10.488 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:10 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3[78174]: 2024-03-04T16:12:10.255+0000 7f1a0cfa97c0 -1 osd.3 121 log_to_monitors true 2024-03-04T16:12:10.976 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:10 smithi083 ceph-mon[64844]: from='osd.3 [v2:172.21.15.83:6826/808233606,v1:172.21.15.83:6827/808233606]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T16:12:10.976 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:10 smithi083 ceph-mon[67383]: from='osd.3 [v2:172.21.15.83:6826/808233606,v1:172.21.15.83:6827/808233606]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T16:12:11.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:10 smithi086 ceph-mon[57919]: from='osd.3 [v2:172.21.15.83:6826/808233606,v1:172.21.15.83:6827/808233606]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T16:12:11.237 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:12:10 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3[78174]: 2024-03-04T16:12:10.975+0000 7f19fe401700 -1 osd.3 121 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:12:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:11 smithi086 ceph-mon[57919]: pgmap v117: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-03-04T16:12:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:11 smithi086 ceph-mon[57919]: from='osd.3 [v2:172.21.15.83:6826/808233606,v1:172.21.15.83:6827/808233606]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-04T16:12:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:11 smithi086 ceph-mon[57919]: osdmap e124: 8 total, 7 up, 8 in 2024-03-04T16:12:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:11 smithi086 ceph-mon[57919]: from='osd.3 [v2:172.21.15.83:6826/808233606,v1:172.21.15.83:6827/808233606]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T16:12:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:12.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:12.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:11 smithi086 ceph-mon[57919]: Health check update: Degraded data redundancy: 83/630 objects degraded (13.175%), 27 pgs degraded (PG_DEGRADED) 2024-03-04T16:12:12.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:11 smithi083 ceph-mon[64844]: pgmap v117: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-03-04T16:12:12.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:11 smithi083 ceph-mon[64844]: from='osd.3 [v2:172.21.15.83:6826/808233606,v1:172.21.15.83:6827/808233606]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-04T16:12:12.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:11 smithi083 ceph-mon[64844]: osdmap e124: 8 total, 7 up, 8 in 2024-03-04T16:12:12.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:11 smithi083 ceph-mon[64844]: from='osd.3 [v2:172.21.15.83:6826/808233606,v1:172.21.15.83:6827/808233606]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T16:12:12.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:12.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:12.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:12.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:12.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:11 smithi083 ceph-mon[64844]: Health check update: Degraded data redundancy: 83/630 objects degraded (13.175%), 27 pgs degraded (PG_DEGRADED) 2024-03-04T16:12:12.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:11 smithi083 ceph-mon[67383]: pgmap v117: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-03-04T16:12:12.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:11 smithi083 ceph-mon[67383]: from='osd.3 [v2:172.21.15.83:6826/808233606,v1:172.21.15.83:6827/808233606]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-04T16:12:12.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:11 smithi083 ceph-mon[67383]: osdmap e124: 8 total, 7 up, 8 in 2024-03-04T16:12:12.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:11 smithi083 ceph-mon[67383]: from='osd.3 [v2:172.21.15.83:6826/808233606,v1:172.21.15.83:6827/808233606]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-04T16:12:12.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:12.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:12.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:12.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:12.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:11 smithi083 ceph-mon[67383]: Health check update: Degraded data redundancy: 83/630 objects degraded (13.175%), 27 pgs degraded (PG_DEGRADED) 2024-03-04T16:12:13.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:12 smithi086 ceph-mon[57919]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:12:13.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:12 smithi086 ceph-mon[57919]: osd.3 [v2:172.21.15.83:6826/808233606,v1:172.21.15.83:6827/808233606] boot 2024-03-04T16:12:13.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:12 smithi086 ceph-mon[57919]: osdmap e125: 8 total, 8 up, 8 in 2024-03-04T16:12:13.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:12 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:12:13.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:12 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:13.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:12 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:13.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:12 smithi086 ceph-mon[57919]: osdmap e126: 8 total, 8 up, 8 in 2024-03-04T16:12:13.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:12 smithi083 ceph-mon[67383]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:12:13.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:12 smithi083 ceph-mon[67383]: osd.3 [v2:172.21.15.83:6826/808233606,v1:172.21.15.83:6827/808233606] boot 2024-03-04T16:12:13.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:12 smithi083 ceph-mon[67383]: osdmap e125: 8 total, 8 up, 8 in 2024-03-04T16:12:13.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:12 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:12:13.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:12 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:13.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:12 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:13.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:12 smithi083 ceph-mon[67383]: osdmap e126: 8 total, 8 up, 8 in 2024-03-04T16:12:13.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:12 smithi083 ceph-mon[64844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:12:13.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:12 smithi083 ceph-mon[64844]: osd.3 [v2:172.21.15.83:6826/808233606,v1:172.21.15.83:6827/808233606] boot 2024-03-04T16:12:13.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:12 smithi083 ceph-mon[64844]: osdmap e125: 8 total, 8 up, 8 in 2024-03-04T16:12:13.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:12 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:12:13.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:12 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:13.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:12 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:13.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:12 smithi083 ceph-mon[64844]: osdmap e126: 8 total, 8 up, 8 in 2024-03-04T16:12:14.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:13 smithi086 ceph-mon[57919]: pgmap v120: 161 pgs: 18 peering, 19 active+undersized+degraded, 29 active+undersized, 95 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 61/630 objects degraded (9.683%) 2024-03-04T16:12:14.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:13 smithi083 ceph-mon[64844]: pgmap v120: 161 pgs: 18 peering, 19 active+undersized+degraded, 29 active+undersized, 95 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 61/630 objects degraded (9.683%) 2024-03-04T16:12:14.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:13 smithi083 ceph-mon[67383]: pgmap v120: 161 pgs: 18 peering, 19 active+undersized+degraded, 29 active+undersized, 95 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 61/630 objects degraded (9.683%) 2024-03-04T16:12:15.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:15 smithi083 ceph-mon[64844]: pgmap v122: 161 pgs: 18 peering, 19 active+undersized+degraded, 29 active+undersized, 95 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 61/630 objects degraded (9.683%) 2024-03-04T16:12:15.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:15 smithi083 ceph-mon[67383]: pgmap v122: 161 pgs: 18 peering, 19 active+undersized+degraded, 29 active+undersized, 95 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 61/630 objects degraded (9.683%) 2024-03-04T16:12:15.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:12:15 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:12:15] "GET /metrics HTTP/1.1" 200 38356 "" "Prometheus/2.33.4" 2024-03-04T16:12:16.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:15 smithi086 ceph-mon[57919]: pgmap v122: 161 pgs: 18 peering, 19 active+undersized+degraded, 29 active+undersized, 95 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 61/630 objects degraded (9.683%) 2024-03-04T16:12:16.488 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:12:16 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:12:16.423+0000 7fe862dc1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-04T16:12:17.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:17 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:17.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:17 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:17.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:17 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:17.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:17 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:12:17.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:17 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:17.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:17 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:17.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:17 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:17 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:17 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:17 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:12:17.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:17 smithi086 ceph-mon[57919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:12:17.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:17 smithi086 ceph-mon[57919]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-04T16:12:17.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:17 smithi086 ceph-mon[57919]: pgmap v123: 161 pgs: 18 peering, 143 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:12:17.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:17 smithi086 ceph-mon[57919]: Health check update: Degraded data redundancy: 61/630 objects degraded (9.683%), 19 pgs degraded (PG_DEGRADED) 2024-03-04T16:12:17.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:17.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:17.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:17.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:12:17.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:17.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:17.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:12:17.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[67383]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:12:17.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[67383]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-04T16:12:17.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[67383]: pgmap v123: 161 pgs: 18 peering, 143 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:12:17.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[67383]: Health check update: Degraded data redundancy: 61/630 objects degraded (9.683%), 19 pgs degraded (PG_DEGRADED) 2024-03-04T16:12:17.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:17.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:17.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:17.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:12:17.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:17.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:17.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:17.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:12:17.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[64844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:12:17.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[64844]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-04T16:12:17.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[64844]: pgmap v123: 161 pgs: 18 peering, 143 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:12:17.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:17 smithi083 ceph-mon[64844]: Health check update: Degraded data redundancy: 61/630 objects degraded (9.683%), 19 pgs degraded (PG_DEGRADED) 2024-03-04T16:12:18.659 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:18 smithi086 ceph-mon[57919]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 61/630 objects degraded (9.683%), 19 pgs degraded) 2024-03-04T16:12:18.659 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:18 smithi086 ceph-mon[57919]: Cluster is now healthy 2024-03-04T16:12:18.659 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:18 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:18.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:18 smithi083 ceph-mon[67383]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 61/630 objects degraded (9.683%), 19 pgs degraded) 2024-03-04T16:12:18.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:18 smithi083 ceph-mon[67383]: Cluster is now healthy 2024-03-04T16:12:18.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:18 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:18.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:18 smithi083 ceph-mon[64844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 61/630 objects degraded (9.683%), 19 pgs degraded) 2024-03-04T16:12:18.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:18 smithi083 ceph-mon[64844]: Cluster is now healthy 2024-03-04T16:12:18.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:18 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:18.984 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:12:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:12:18.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-03-04T16:12:18.984 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:12:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:12:18.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"smithi083:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.83\", device_class=\"ssd\", hostname=\"smithi083\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.83\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-04T16:12:19.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:19 smithi086 ceph-mon[57919]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:12:19.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:19 smithi083 ceph-mon[67383]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:12:19.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:19 smithi083 ceph-mon[64844]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:12:20.128 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:12:20.770 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:12:20.770 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (5m) 9s ago 10m 24.9M - 0.23.0 ba2b418f427c 3c6dc3779a0e 2024-03-04T16:12:20.770 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (4m) 2m ago 9m 46.6M - 8.3.5 dad864ee21e9 fb754d58c53f 2024-03-04T16:12:20.770 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 running (3m) 9s ago 9m 53.6M - 3.5 e1d6a67b021e d58e71ccf9db 2024-03-04T16:12:20.770 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443,9283,8765 running (3m) 2m ago 14m 448M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c33100cb8a8 2024-03-04T16:12:20.770 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:8443,9283,8765 running (4m) 9s ago 16m 506M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 164dd93609ee 2024-03-04T16:12:20.770 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (3m) 9s ago 16m 48.8M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c173a2442197 2024-03-04T16:12:20.770 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (2m) 2m ago 14m 16.7M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 71557985ed5a 2024-03-04T16:12:20.771 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (3m) 9s ago 15m 40.6M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e eff1982ba275 2024-03-04T16:12:20.771 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (5m) 9s ago 10m 17.9M - 1.3.1 1dbe0e931976 9bf54b460640 2024-03-04T16:12:20.771 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (4m) 2m ago 10m 16.0M - 1.3.1 1dbe0e931976 b72dd80b2588 2024-03-04T16:12:20.771 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (119s) 9s ago 14m 44.4M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e f7ecdc5b1520 2024-03-04T16:12:20.771 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (84s) 9s ago 13m 46.6M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 2c648d9a2b38 2024-03-04T16:12:20.771 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (49s) 9s ago 13m 53.0M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c971c1f3f0a 2024-03-04T16:12:20.771 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (14s) 9s ago 13m 11.6M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 8991d467d41c 2024-03-04T16:12:20.771 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (12m) 2m ago 12m 53.9M 2992M 17.2.0 e1d6a67b021e b3a97f83f1c7 2024-03-04T16:12:20.771 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (12m) 2m ago 12m 50.7M 2992M 17.2.0 e1d6a67b021e b82ca42b7353 2024-03-04T16:12:20.771 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (12m) 2m ago 12m 48.8M 2992M 17.2.0 e1d6a67b021e 18cedf14321d 2024-03-04T16:12:20.771 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (11m) 2m ago 11m 54.5M 2992M 17.2.0 e1d6a67b021e 390e1798c738 2024-03-04T16:12:20.771 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 running (3m) 2m ago 10m 58.2M - 2.33.4 514e6a882f6e b563a246eeac 2024-03-04T16:12:20.771 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (9m) 9s ago 9m 74.5M - 17.2.0 e1d6a67b021e 65f14042f354 2024-03-04T16:12:20.771 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (9m) 2m ago 9m 74.9M - 17.2.0 e1d6a67b021e 78175cc6dc5d 2024-03-04T16:12:20.771 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (9m) 9s ago 9m 72.6M - 17.2.0 e1d6a67b021e 01dcf858bd62 2024-03-04T16:12:20.772 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (9m) 2m ago 9m 76.7M - 17.2.0 e1d6a67b021e 8f70a5ce9f0d 2024-03-04T16:12:21.160 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:12:21.161 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:12:21.161 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 3 2024-03-04T16:12:21.161 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:12:21.161 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:12:21.161 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:12:21.161 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:12:21.161 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:12:21.161 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-04T16:12:21.161 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 4 2024-03-04T16:12:21.161 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:12:21.161 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:12:21.161 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-04T16:12:21.161 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:12:21.162 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:12:21.162 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-03-04T16:12:21.162 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 9 2024-03-04T16:12:21.162 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:12:21.162 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:12:21.488 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:12:21.488 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:12:21.488 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:12:21.488 INFO:teuthology.orchestra.run.smithi083.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:12:21.488 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [ 2024-03-04T16:12:21.488 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr", 2024-03-04T16:12:21.488 INFO:teuthology.orchestra.run.smithi083.stdout: "mon" 2024-03-04T16:12:21.488 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-04T16:12:21.489 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "9/23 daemons upgraded", 2024-03-04T16:12:21.489 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "Currently upgrading osd daemons", 2024-03-04T16:12:21.489 INFO:teuthology.orchestra.run.smithi083.stdout: "is_paused": false 2024-03-04T16:12:21.489 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:12:21.880 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_OK 2024-03-04T16:12:22.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:21 smithi086 ceph-mon[57919]: from='client.44376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:22.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:21 smithi086 ceph-mon[57919]: from='client.54313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:22.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:21 smithi086 ceph-mon[57919]: from='client.44388 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:22.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:21 smithi086 ceph-mon[57919]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:12:22.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:21 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/2862803418' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:22.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:21 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/3386972312' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:12:22.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:21 smithi083 ceph-mon[64844]: from='client.44376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:22.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:21 smithi083 ceph-mon[64844]: from='client.54313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:22.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:21 smithi083 ceph-mon[64844]: from='client.44388 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:22.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:21 smithi083 ceph-mon[64844]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:12:22.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:21 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/2862803418' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:22.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:21 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/3386972312' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:12:22.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:21 smithi083 ceph-mon[67383]: from='client.44376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:22.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:21 smithi083 ceph-mon[67383]: from='client.54313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:22.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:21 smithi083 ceph-mon[67383]: from='client.44388 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:22.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:21 smithi083 ceph-mon[67383]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:12:22.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:21 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/2862803418' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:22.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:21 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/3386972312' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:12:23.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:22 smithi086 ceph-mon[57919]: from='client.54331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:23.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:22 smithi083 ceph-mon[67383]: from='client.54331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:23.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:22 smithi083 ceph-mon[64844]: from='client.54331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:24.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:23 smithi086 ceph-mon[57919]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s 2024-03-04T16:12:24.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:23 smithi083 ceph-mon[67383]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s 2024-03-04T16:12:24.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:23 smithi083 ceph-mon[64844]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s 2024-03-04T16:12:25.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:25 smithi083 ceph-mon[67383]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 258 B/s rd, 0 op/s 2024-03-04T16:12:25.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:25 smithi083 ceph-mon[64844]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 258 B/s rd, 0 op/s 2024-03-04T16:12:25.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:12:25 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:12:25] "GET /metrics HTTP/1.1" 200 38356 "" "Prometheus/2.33.4" 2024-03-04T16:12:26.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:25 smithi086 ceph-mon[57919]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 258 B/s rd, 0 op/s 2024-03-04T16:12:28.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:27 smithi086 ceph-mon[57919]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-04T16:12:28.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:28.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:28.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:28.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:27 smithi083 ceph-mon[67383]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-04T16:12:28.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:28.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:28.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:28.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:27 smithi083 ceph-mon[64844]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-04T16:12:28.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:28.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:28.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:29.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:28 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:29.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:28 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:29.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:28 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:30.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:29 smithi086 ceph-mon[57919]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:12:30.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:29 smithi083 ceph-mon[67383]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:12:30.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:29 smithi083 ceph-mon[64844]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:12:32.200 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:31 smithi086 ceph-mon[57919]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:12:32.200 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:31 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:12:32.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:31 smithi083 ceph-mon[67383]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:12:32.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:31 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:12:32.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:31 smithi083 ceph-mon[64844]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:12:32.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:31 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:12:33.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:32 smithi086 ceph-mon[57919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:12:33.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:32 smithi086 ceph-mon[57919]: Upgrade: osd.4 is safe to restart 2024-03-04T16:12:33.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:32 smithi083 ceph-mon[67383]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:12:33.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:32 smithi083 ceph-mon[67383]: Upgrade: osd.4 is safe to restart 2024-03-04T16:12:33.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:32 smithi083 ceph-mon[64844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-04T16:12:33.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:32 smithi083 ceph-mon[64844]: Upgrade: osd.4 is safe to restart 2024-03-04T16:12:34.212 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:33 smithi086 ceph-mon[57919]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:12:34.212 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:33 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:34.212 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:33 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T16:12:34.212 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:33 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:34.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:33 smithi083 ceph-mon[64844]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:12:34.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:33 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:34.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:33 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T16:12:34.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:33 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:34.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:33 smithi083 ceph-mon[67383]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:12:34.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:33 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:34.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:33 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T16:12:34.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:33 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:35.147 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:34 smithi086 ceph-mon[57919]: Upgrade: Updating osd.4 2024-03-04T16:12:35.147 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:34 smithi086 ceph-mon[57919]: Deploying daemon osd.4 on smithi086 2024-03-04T16:12:35.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:34 smithi083 ceph-mon[64844]: Upgrade: Updating osd.4 2024-03-04T16:12:35.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:34 smithi083 ceph-mon[64844]: Deploying daemon osd.4 on smithi086 2024-03-04T16:12:35.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:34 smithi083 ceph-mon[67383]: Upgrade: Updating osd.4 2024-03-04T16:12:35.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:34 smithi083 ceph-mon[67383]: Deploying daemon osd.4 on smithi086 2024-03-04T16:12:35.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:35 smithi083 ceph-mon[64844]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:12:35.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:35 smithi083 ceph-mon[67383]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:12:35.989 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:12:35 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:12:35] "GET /metrics HTTP/1.1" 200 38513 "" "Prometheus/2.33.4" 2024-03-04T16:12:36.065 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:35 smithi086 ceph-mon[57919]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:12:36.670 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:36 smithi086 systemd[1]: Stopping Ceph osd.4 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:12:36.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:36 smithi086 ceph-mon[57919]: osd.4 marked itself down and dead 2024-03-04T16:12:36.984 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:36 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[28935]: 2024-03-04T16:12:36.669+0000 7f900393f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:12:36.984 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:36 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[28935]: 2024-03-04T16:12:36.669+0000 7f900393f700 -1 osd.4 126 *** Got signal Terminated *** 2024-03-04T16:12:36.984 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:36 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[28935]: 2024-03-04T16:12:36.669+0000 7f900393f700 -1 osd.4 126 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:12:37.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:36 smithi083 ceph-mon[64844]: osd.4 marked itself down and dead 2024-03-04T16:12:37.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:36 smithi083 ceph-mon[67383]: osd.4 marked itself down and dead 2024-03-04T16:12:37.520 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:37 smithi086 podman[60744]: 2024-03-04 16:12:37.259264543 +0000 UTC m=+0.707205963 container died b3a97f83f1c72bb77003e015d5b53f424c0e27d7d403d4f788423f2ad7b0f618 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, version=8, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, release=754, io.openshift.expose-services=, RELEASE=HEAD, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, build-date=2022-05-03T08:36:31.336870) 2024-03-04T16:12:37.903 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:37 smithi086 podman[60744]: 2024-03-04 16:12:37.648148395 +0000 UTC m=+1.096089826 container cleanup b3a97f83f1c72bb77003e015d5b53f424c0e27d7d403d4f788423f2ad7b0f618 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, GIT_BRANCH=HEAD, io.openshift.expose-services=, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , vcs-type=git, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD) 2024-03-04T16:12:37.903 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:37 smithi086 bash[60744]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4 2024-03-04T16:12:38.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:37 smithi086 ceph-mon[57919]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:12:38.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:37 smithi086 ceph-mon[57919]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:12:38.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:37 smithi086 ceph-mon[57919]: osdmap e127: 8 total, 7 up, 8 in 2024-03-04T16:12:38.234 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:37 smithi086 podman[60771]: 2024-03-04 16:12:37.903888065 +0000 UTC m=+0.645990591 container remove b3a97f83f1c72bb77003e015d5b53f424c0e27d7d403d4f788423f2ad7b0f618 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4, architecture=x86_64, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, io.buildah.version=1.19.8, vcs-type=git, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, vendor=Red Hat, Inc., release=754, distribution-scope=public, com.redhat.component=centos-stream-container) 2024-03-04T16:12:38.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:37 smithi083 ceph-mon[64844]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:12:38.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:37 smithi083 ceph-mon[64844]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:12:38.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:37 smithi083 ceph-mon[64844]: osdmap e127: 8 total, 7 up, 8 in 2024-03-04T16:12:38.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:37 smithi083 ceph-mon[67383]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:12:38.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:37 smithi083 ceph-mon[67383]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:12:38.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:37 smithi083 ceph-mon[67383]: osdmap e127: 8 total, 7 up, 8 in 2024-03-04T16:12:38.557 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:38 smithi086 podman[60862]: 2024-03-04 16:12:38.157692587 +0000 UTC m=+0.017119235 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:12:38.557 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:38 smithi086 podman[60862]: 2024-03-04 16:12:38.296832416 +0000 UTC m=+0.156259057 container create c184829772c2a322bd7d0d5060b4073df96dfc3743c73eb106910bc6225fe0df (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:12:38.557 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:38 smithi086 podman[60862]: 2024-03-04 16:12:38.499034075 +0000 UTC m=+0.358460723 container init c184829772c2a322bd7d0d5060b4073df96dfc3743c73eb106910bc6225fe0df (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-04T16:12:38.557 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:38 smithi086 podman[60862]: 2024-03-04 16:12:38.502068244 +0000 UTC m=+0.361494890 container start c184829772c2a322bd7d0d5060b4073df96dfc3743c73eb106910bc6225fe0df (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=) 2024-03-04T16:12:38.557 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:38 smithi086 podman[60862]: 2024-03-04 16:12:38.557403644 +0000 UTC m=+0.416830302 container attach c184829772c2a322bd7d0d5060b4073df96dfc3743c73eb106910bc6225fe0df (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-04T16:12:38.917 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:38 smithi086 podman[60862]: 2024-03-04 16:12:38.791178867 +0000 UTC m=+0.650605508 container died c184829772c2a322bd7d0d5060b4073df96dfc3743c73eb106910bc6225fe0df (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:12:39.233 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:38 smithi086 ceph-mon[57919]: osdmap e128: 8 total, 7 up, 8 in 2024-03-04T16:12:39.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:38 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:39.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:38 smithi083 ceph-mon[64844]: osdmap e128: 8 total, 7 up, 8 in 2024-03-04T16:12:39.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:38 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:39.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:38 smithi083 ceph-mon[67383]: osdmap e128: 8 total, 7 up, 8 in 2024-03-04T16:12:39.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:38 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:39.590 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:39 smithi086 podman[60888]: 2024-03-04 16:12:39.378110397 +0000 UTC m=+0.575670099 container remove c184829772c2a322bd7d0d5060b4073df96dfc3743c73eb106910bc6225fe0df (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-03-04T16:12:39.590 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:39 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-04T16:12:39.590 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:39 smithi086 systemd[1]: Stopped Ceph osd.4 for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:12:39.590 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:39 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.4.service: Consumed 8.579s CPU time. 2024-03-04T16:12:39.918 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:39 smithi086 systemd[1]: Starting Ceph osd.4 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:12:39.918 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:39 smithi086 podman[60993]: 2024-03-04 16:12:39.708684228 +0000 UTC m=+0.017552039 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:12:39.918 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:39 smithi086 podman[60993]: 2024-03-04 16:12:39.812332614 +0000 UTC m=+0.121200427 container create 561af8d45eae56d89624f753770a99b44fcccdbb05e02372ec446c9f883b339f (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-activate, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, ceph=True) 2024-03-04T16:12:40.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:39 smithi086 ceph-mon[57919]: pgmap v136: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:12:40.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:39 smithi086 ceph-mon[57919]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-03-04T16:12:40.234 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:40 smithi086 podman[60993]: 2024-03-04 16:12:40.031610161 +0000 UTC m=+0.340477973 container init 561af8d45eae56d89624f753770a99b44fcccdbb05e02372ec446c9f883b339f (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:12:40.235 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:40 smithi086 podman[60993]: 2024-03-04 16:12:40.034799676 +0000 UTC m=+0.343667492 container start 561af8d45eae56d89624f753770a99b44fcccdbb05e02372ec446c9f883b339f (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-04T16:12:40.235 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:40 smithi086 podman[60993]: 2024-03-04 16:12:40.131537422 +0000 UTC m=+0.440405247 container attach 561af8d45eae56d89624f753770a99b44fcccdbb05e02372ec446c9f883b339f (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-activate, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222) 2024-03-04T16:12:40.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:39 smithi083 ceph-mon[64844]: pgmap v136: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:12:40.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:39 smithi083 ceph-mon[64844]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-03-04T16:12:40.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:39 smithi083 ceph-mon[67383]: pgmap v136: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:12:40.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:39 smithi083 ceph-mon[67383]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-03-04T16:12:40.734 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:40 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-activate[61009]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-04T16:12:40.734 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:40 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-activate[61009]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-04T16:12:40.734 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:40 smithi086 bash[60993]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-04T16:12:40.734 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:40 smithi086 bash[60993]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-04T16:12:40.734 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:40 smithi086 bash[60993]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-04T16:12:40.734 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:40 smithi086 bash[60993]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-04T16:12:40.735 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:40 smithi086 bash[60993]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-04T16:12:40.735 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:40 smithi086 bash[60993]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-04T16:12:40.735 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:40 smithi086 bash[60993]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-04T16:12:40.735 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:40 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-activate[61009]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-04T16:12:40.735 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:40 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-activate[61009]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-04T16:12:40.735 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:40 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-activate[61009]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-04T16:12:40.735 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:40 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-activate[61009]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-04T16:12:40.735 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:40 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-activate[61009]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-04T16:12:40.735 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:40 smithi086 podman[60993]: 2024-03-04 16:12:40.498783476 +0000 UTC m=+0.807651291 container died 561af8d45eae56d89624f753770a99b44fcccdbb05e02372ec446c9f883b339f (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-activate, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-04T16:12:41.327 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:41 smithi086 podman[61126]: 2024-03-04 16:12:41.060048332 +0000 UTC m=+0.556188678 container remove 561af8d45eae56d89624f753770a99b44fcccdbb05e02372ec446c9f883b339f (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True) 2024-03-04T16:12:41.328 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:41 smithi086 podman[61194]: 2024-03-04 16:12:41.174883461 +0000 UTC m=+0.018806908 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:12:41.328 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:41 smithi086 podman[61194]: 2024-03-04 16:12:41.328062174 +0000 UTC m=+0.171985621 container create 912383028c724e2322ef8f858df94d326c1ce58da8b0d812c62f000d30e6cf81 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:12:41.638 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:41 smithi086 podman[61194]: 2024-03-04 16:12:41.539195064 +0000 UTC m=+0.383118530 container init 912383028c724e2322ef8f858df94d326c1ce58da8b0d812c62f000d30e6cf81 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:12:41.639 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:41 smithi086 podman[61194]: 2024-03-04 16:12:41.542336034 +0000 UTC m=+0.386259487 container start 912383028c724e2322ef8f858df94d326c1ce58da8b0d812c62f000d30e6cf81 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:12:41.639 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:41 smithi086 ceph-osd[61213]: -- 172.21.15.86:0/3124784212 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562bb332f680 con 0x562bb32f9400 2024-03-04T16:12:41.921 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:41 smithi086 bash[61194]: 912383028c724e2322ef8f858df94d326c1ce58da8b0d812c62f000d30e6cf81 2024-03-04T16:12:41.922 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:41 smithi086 systemd[1]: Started Ceph osd.4 for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:12:42.234 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:42 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[61209]: 2024-03-04T16:12:42.089+0000 7f374747a7c0 -1 Falling back to public interface 2024-03-04T16:12:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:41 smithi086 ceph-mon[57919]: pgmap v137: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:12:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:42.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:41 smithi083 ceph-mon[64844]: pgmap v137: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:12:42.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:42.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:42.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:42.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:42.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:41 smithi083 ceph-mon[67383]: pgmap v137: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:12:42.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:42.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:42.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:42.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:42 smithi086 ceph-mon[57919]: Health check failed: Degraded data redundancy: 106/630 objects degraded (16.825%), 26 pgs degraded (PG_DEGRADED) 2024-03-04T16:12:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:42 smithi086 ceph-mon[57919]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-03-04T16:12:43.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:42 smithi083 ceph-mon[64844]: Health check failed: Degraded data redundancy: 106/630 objects degraded (16.825%), 26 pgs degraded (PG_DEGRADED) 2024-03-04T16:12:43.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:42 smithi083 ceph-mon[64844]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-03-04T16:12:43.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:42 smithi083 ceph-mon[67383]: Health check failed: Degraded data redundancy: 106/630 objects degraded (16.825%), 26 pgs degraded (PG_DEGRADED) 2024-03-04T16:12:43.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:42 smithi083 ceph-mon[67383]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-03-04T16:12:44.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:43 smithi086 ceph-mon[57919]: pgmap v138: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 106/630 objects degraded (16.825%) 2024-03-04T16:12:44.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:43 smithi083 ceph-mon[64844]: pgmap v138: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 106/630 objects degraded (16.825%) 2024-03-04T16:12:44.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:43 smithi083 ceph-mon[67383]: pgmap v138: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 106/630 objects degraded (16.825%) 2024-03-04T16:12:45.019 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:44 smithi086 ceph-mon[57919]: from='osd.4 [v2:172.21.15.86:6800/4006112720,v1:172.21.15.86:6801/4006112720]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T16:12:45.019 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:44 smithi086 ceph-mon[57919]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T16:12:45.019 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[61209]: 2024-03-04T16:12:44.797+0000 7f374747a7c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-03-04T16:12:45.020 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:44 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[61209]: 2024-03-04T16:12:44.828+0000 7f374747a7c0 -1 osd.4 126 log_to_monitors true 2024-03-04T16:12:45.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:44 smithi083 ceph-mon[64844]: from='osd.4 [v2:172.21.15.86:6800/4006112720,v1:172.21.15.86:6801/4006112720]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T16:12:45.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:44 smithi083 ceph-mon[64844]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T16:12:45.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:44 smithi083 ceph-mon[67383]: from='osd.4 [v2:172.21.15.86:6800/4006112720,v1:172.21.15.86:6801/4006112720]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T16:12:45.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:44 smithi083 ceph-mon[67383]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T16:12:45.734 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:12:45 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[61209]: 2024-03-04T16:12:45.455+0000 7f37348ca700 -1 osd.4 126 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:12:45.982 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:12:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:12:45] "GET /metrics HTTP/1.1" 200 38508 "" "Prometheus/2.33.4" 2024-03-04T16:12:46.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:45 smithi086 ceph-mon[57919]: pgmap v139: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 106/630 objects degraded (16.825%) 2024-03-04T16:12:46.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:45 smithi086 ceph-mon[57919]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-04T16:12:46.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:45 smithi086 ceph-mon[57919]: osdmap e129: 8 total, 7 up, 8 in 2024-03-04T16:12:46.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:45 smithi086 ceph-mon[57919]: from='osd.4 [v2:172.21.15.86:6800/4006112720,v1:172.21.15.86:6801/4006112720]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:12:46.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:45 smithi086 ceph-mon[57919]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:12:46.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:45 smithi086 ceph-mon[57919]: from='osd.4 ' entity='osd.4' 2024-03-04T16:12:46.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:45 smithi083 ceph-mon[67383]: pgmap v139: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 106/630 objects degraded (16.825%) 2024-03-04T16:12:46.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:45 smithi083 ceph-mon[67383]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-04T16:12:46.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:45 smithi083 ceph-mon[67383]: osdmap e129: 8 total, 7 up, 8 in 2024-03-04T16:12:46.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:45 smithi083 ceph-mon[67383]: from='osd.4 [v2:172.21.15.86:6800/4006112720,v1:172.21.15.86:6801/4006112720]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:12:46.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:45 smithi083 ceph-mon[67383]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:12:46.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:45 smithi083 ceph-mon[67383]: from='osd.4 ' entity='osd.4' 2024-03-04T16:12:46.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:45 smithi083 ceph-mon[64844]: pgmap v139: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 106/630 objects degraded (16.825%) 2024-03-04T16:12:46.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:45 smithi083 ceph-mon[64844]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-04T16:12:46.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:45 smithi083 ceph-mon[64844]: osdmap e129: 8 total, 7 up, 8 in 2024-03-04T16:12:46.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:45 smithi083 ceph-mon[64844]: from='osd.4 [v2:172.21.15.86:6800/4006112720,v1:172.21.15.86:6801/4006112720]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:12:46.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:45 smithi083 ceph-mon[64844]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:12:46.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:45 smithi083 ceph-mon[64844]: from='osd.4 ' entity='osd.4' 2024-03-04T16:12:47.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:47 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:47.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:47 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:47.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:47 smithi086 ceph-mon[57919]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:12:47.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:47 smithi086 ceph-mon[57919]: osd.4 [v2:172.21.15.86:6800/4006112720,v1:172.21.15.86:6801/4006112720] boot 2024-03-04T16:12:47.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:47 smithi086 ceph-mon[57919]: osdmap e130: 8 total, 8 up, 8 in 2024-03-04T16:12:47.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:47 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:12:47.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:47 smithi086 ceph-mon[57919]: pgmap v142: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 106/630 objects degraded (16.825%) 2024-03-04T16:12:47.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:47 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:47.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:47 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:47.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:47 smithi083 ceph-mon[64844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:12:47.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:47 smithi083 ceph-mon[64844]: osd.4 [v2:172.21.15.86:6800/4006112720,v1:172.21.15.86:6801/4006112720] boot 2024-03-04T16:12:47.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:47 smithi083 ceph-mon[64844]: osdmap e130: 8 total, 8 up, 8 in 2024-03-04T16:12:47.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:47 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:12:47.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:47 smithi083 ceph-mon[64844]: pgmap v142: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 106/630 objects degraded (16.825%) 2024-03-04T16:12:47.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:47 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:47.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:47 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:47.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:47 smithi083 ceph-mon[67383]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:12:47.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:47 smithi083 ceph-mon[67383]: osd.4 [v2:172.21.15.86:6800/4006112720,v1:172.21.15.86:6801/4006112720] boot 2024-03-04T16:12:47.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:47 smithi083 ceph-mon[67383]: osdmap e130: 8 total, 8 up, 8 in 2024-03-04T16:12:47.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:47 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:12:47.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:47 smithi083 ceph-mon[67383]: pgmap v142: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 106/630 objects degraded (16.825%) 2024-03-04T16:12:48.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:48 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:48.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:48 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:48.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:48 smithi086 ceph-mon[57919]: osdmap e131: 8 total, 8 up, 8 in 2024-03-04T16:12:48.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:48 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:48.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:48 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:48.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:48 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:48.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:48 smithi083 ceph-mon[64844]: osdmap e131: 8 total, 8 up, 8 in 2024-03-04T16:12:48.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:48 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:48.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:48 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:48.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:48 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:48.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:48 smithi083 ceph-mon[67383]: osdmap e131: 8 total, 8 up, 8 in 2024-03-04T16:12:48.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:48 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:49.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:49 smithi086 ceph-mon[57919]: pgmap v144: 161 pgs: 18 active+undersized+degraded, 27 active+undersized, 116 active+clean; 585 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 58/630 objects degraded (9.206%) 2024-03-04T16:12:49.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:49 smithi083 ceph-mon[64844]: pgmap v144: 161 pgs: 18 active+undersized+degraded, 27 active+undersized, 116 active+clean; 585 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 58/630 objects degraded (9.206%) 2024-03-04T16:12:49.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:49 smithi083 ceph-mon[67383]: pgmap v144: 161 pgs: 18 active+undersized+degraded, 27 active+undersized, 116 active+clean; 585 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 58/630 objects degraded (9.206%) 2024-03-04T16:12:50.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:50 smithi086 ceph-mon[57919]: Health check update: Degraded data redundancy: 58/630 objects degraded (9.206%), 18 pgs degraded (PG_DEGRADED) 2024-03-04T16:12:50.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:50 smithi083 ceph-mon[64844]: Health check update: Degraded data redundancy: 58/630 objects degraded (9.206%), 18 pgs degraded (PG_DEGRADED) 2024-03-04T16:12:50.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:50 smithi083 ceph-mon[67383]: Health check update: Degraded data redundancy: 58/630 objects degraded (9.206%), 18 pgs degraded (PG_DEGRADED) 2024-03-04T16:12:51.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:12:50 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:12:50.776+0000 7fe862dc1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-03-04T16:12:51.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:51 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:51.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:51 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:51.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:51 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:51.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:51 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:12:51.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:51 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:51.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:51 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:51.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:51 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:51.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:51 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:51.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:51 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:51.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:51 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:12:51.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:51 smithi086 ceph-mon[57919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:12:51.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:51 smithi086 ceph-mon[57919]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-03-04T16:12:51.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:51 smithi086 ceph-mon[57919]: pgmap v145: 161 pgs: 8 active+undersized+degraded, 14 active+undersized, 139 active+clean; 585 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 30/630 objects degraded (4.762%) 2024-03-04T16:12:51.987 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:51.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:51.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:51.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:12:51.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:51.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:51.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:51.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:51.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:51.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:12:51.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[67383]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:12:51.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[67383]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-03-04T16:12:51.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[67383]: pgmap v145: 161 pgs: 8 active+undersized+degraded, 14 active+undersized, 139 active+clean; 585 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 30/630 objects degraded (4.762%) 2024-03-04T16:12:51.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:51.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:51.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:51.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:12:51.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:51.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:51.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:51.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:51.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:51.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:12:51.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[64844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:12:51.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[64844]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-03-04T16:12:51.990 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:51 smithi083 ceph-mon[64844]: pgmap v145: 161 pgs: 8 active+undersized+degraded, 14 active+undersized, 139 active+clean; 585 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 30/630 objects degraded (4.762%) 2024-03-04T16:12:52.236 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:12:52.882 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:12:52.882 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (5m) 41s ago 10m 24.9M - 0.23.0 ba2b418f427c 3c6dc3779a0e 2024-03-04T16:12:52.882 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (5m) 6s ago 10m 54.6M - 8.3.5 dad864ee21e9 fb754d58c53f 2024-03-04T16:12:52.883 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 running (4m) 41s ago 9m 53.6M - 3.5 e1d6a67b021e d58e71ccf9db 2024-03-04T16:12:52.883 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443,9283,8765 running (4m) 6s ago 15m 451M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c33100cb8a8 2024-03-04T16:12:52.883 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:8443,9283,8765 running (4m) 41s ago 16m 506M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 164dd93609ee 2024-03-04T16:12:52.883 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (4m) 41s ago 16m 48.8M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c173a2442197 2024-03-04T16:12:52.883 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (3m) 6s ago 15m 38.0M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 71557985ed5a 2024-03-04T16:12:52.883 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (3m) 41s ago 15m 40.6M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e eff1982ba275 2024-03-04T16:12:52.883 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (5m) 41s ago 11m 17.9M - 1.3.1 1dbe0e931976 9bf54b460640 2024-03-04T16:12:52.883 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (5m) 6s ago 11m 15.5M - 1.3.1 1dbe0e931976 b72dd80b2588 2024-03-04T16:12:52.883 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (2m) 41s ago 14m 44.4M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e f7ecdc5b1520 2024-03-04T16:12:52.883 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (116s) 41s ago 14m 46.6M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 2c648d9a2b38 2024-03-04T16:12:52.883 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (81s) 41s ago 14m 53.0M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c971c1f3f0a 2024-03-04T16:12:52.884 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (46s) 41s ago 13m 11.6M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 8991d467d41c 2024-03-04T16:12:52.884 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (11s) 6s ago 13m 11.6M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 912383028c72 2024-03-04T16:12:52.884 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (12m) 6s ago 12m 55.9M 2992M 17.2.0 e1d6a67b021e b82ca42b7353 2024-03-04T16:12:52.884 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (12m) 6s ago 12m 54.0M 2992M 17.2.0 e1d6a67b021e 18cedf14321d 2024-03-04T16:12:52.884 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (12m) 6s ago 12m 60.0M 2992M 17.2.0 e1d6a67b021e 390e1798c738 2024-03-04T16:12:52.884 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 running (4m) 6s ago 11m 71.5M - 2.33.4 514e6a882f6e b563a246eeac 2024-03-04T16:12:52.884 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (9m) 41s ago 9m 74.5M - 17.2.0 e1d6a67b021e 65f14042f354 2024-03-04T16:12:52.884 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (9m) 6s ago 9m 75.6M - 17.2.0 e1d6a67b021e 78175cc6dc5d 2024-03-04T16:12:52.884 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (9m) 41s ago 9m 72.6M - 17.2.0 e1d6a67b021e 01dcf858bd62 2024-03-04T16:12:52.884 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (9m) 6s ago 9m 77.5M - 17.2.0 e1d6a67b021e 8f70a5ce9f0d 2024-03-04T16:12:53.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:52 smithi086 ceph-mon[57919]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 30/630 objects degraded (4.762%), 8 pgs degraded) 2024-03-04T16:12:53.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:52 smithi086 ceph-mon[57919]: Cluster is now healthy 2024-03-04T16:12:53.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:52 smithi083 ceph-mon[67383]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 30/630 objects degraded (4.762%), 8 pgs degraded) 2024-03-04T16:12:53.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:52 smithi083 ceph-mon[67383]: Cluster is now healthy 2024-03-04T16:12:53.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:52 smithi083 ceph-mon[64844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 30/630 objects degraded (4.762%), 8 pgs degraded) 2024-03-04T16:12:53.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:52 smithi083 ceph-mon[64844]: Cluster is now healthy 2024-03-04T16:12:53.279 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:12:53.280 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:12:53.280 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 3 2024-03-04T16:12:53.280 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:12:53.280 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:12:53.280 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:12:53.280 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:12:53.280 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:12:53.280 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-04T16:12:53.280 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 5 2024-03-04T16:12:53.280 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:12:53.281 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:12:53.281 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-04T16:12:53.281 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:12:53.281 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:12:53.281 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-04T16:12:53.281 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 10 2024-03-04T16:12:53.281 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:12:53.281 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:12:53.605 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:12:53.605 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:12:53.605 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:12:53.606 INFO:teuthology.orchestra.run.smithi083.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:12:53.606 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [ 2024-03-04T16:12:53.606 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr", 2024-03-04T16:12:53.606 INFO:teuthology.orchestra.run.smithi083.stdout: "mon" 2024-03-04T16:12:53.606 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-04T16:12:53.606 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "10/23 daemons upgraded", 2024-03-04T16:12:53.606 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "Currently upgrading osd daemons", 2024-03-04T16:12:53.606 INFO:teuthology.orchestra.run.smithi083.stdout: "is_paused": false 2024-03-04T16:12:53.606 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:12:53.992 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_OK 2024-03-04T16:12:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:53 smithi086 ceph-mon[57919]: from='client.44415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:53 smithi086 ceph-mon[57919]: from='client.54352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:53 smithi086 ceph-mon[57919]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:12:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:53 smithi086 ceph-mon[57919]: from='client.44427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:54.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:53 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/960872126' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:54.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:53 smithi083 ceph-mon[64844]: from='client.44415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:54.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:53 smithi083 ceph-mon[64844]: from='client.54352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:54.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:53 smithi083 ceph-mon[64844]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:12:54.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:53 smithi083 ceph-mon[64844]: from='client.44427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:54.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:53 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/960872126' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:54.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:53 smithi083 ceph-mon[67383]: from='client.44415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:54.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:53 smithi083 ceph-mon[67383]: from='client.54352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:54.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:53 smithi083 ceph-mon[67383]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:12:54.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:53 smithi083 ceph-mon[67383]: from='client.44427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:54.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:53 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/960872126' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:12:55.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:54 smithi086 ceph-mon[57919]: from='client.44439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:55.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:54 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/298867704' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:12:55.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:54 smithi083 ceph-mon[64844]: from='client.44439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:55.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:54 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/298867704' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:12:55.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:54 smithi083 ceph-mon[67383]: from='client.44439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:12:55.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:54 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/298867704' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:12:55.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:55 smithi083 ceph-mon[64844]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 610 B/s rd, 0 op/s 2024-03-04T16:12:55.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:55 smithi083 ceph-mon[67383]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 610 B/s rd, 0 op/s 2024-03-04T16:12:55.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:12:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:12:55] "GET /metrics HTTP/1.1" 200 38508 "" "Prometheus/2.33.4" 2024-03-04T16:12:56.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:55 smithi086 ceph-mon[57919]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 610 B/s rd, 0 op/s 2024-03-04T16:12:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:57 smithi086 ceph-mon[57919]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:12:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:58.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:58.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:57 smithi083 ceph-mon[64844]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:12:58.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:58.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:58.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:57 smithi083 ceph-mon[67383]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:12:58.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:12:58.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:00.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:59 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:00.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:12:59 smithi086 ceph-mon[57919]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-03-04T16:13:00.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:59 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:00.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:12:59 smithi083 ceph-mon[64844]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-03-04T16:13:00.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:59 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:00.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:12:59 smithi083 ceph-mon[67383]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-03-04T16:13:02.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:01 smithi086 ceph-mon[57919]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 39 op/s 2024-03-04T16:13:02.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:01 smithi083 ceph-mon[67383]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 39 op/s 2024-03-04T16:13:02.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:01 smithi083 ceph-mon[64844]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 39 op/s 2024-03-04T16:13:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:02 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:02 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:03.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:02 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:04.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:03 smithi086 ceph-mon[57919]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-04T16:13:04.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:03 smithi083 ceph-mon[67383]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-04T16:13:04.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:03 smithi083 ceph-mon[64844]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-04T16:13:05.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:05 smithi083 ceph-mon[67383]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-04T16:13:05.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:05 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:13:05.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:05 smithi083 ceph-mon[64844]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-04T16:13:05.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:05 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:13:05.989 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:13:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:13:05] "GET /metrics HTTP/1.1" 200 38514 "" "Prometheus/2.33.4" 2024-03-04T16:13:06.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:05 smithi086 ceph-mon[57919]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-04T16:13:06.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:05 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:13:07.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:06 smithi086 ceph-mon[57919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:13:07.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:06 smithi086 ceph-mon[57919]: Upgrade: osd.5 is safe to restart 2024-03-04T16:13:07.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:06 smithi083 ceph-mon[67383]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:13:07.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:06 smithi083 ceph-mon[67383]: Upgrade: osd.5 is safe to restart 2024-03-04T16:13:07.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:06 smithi083 ceph-mon[64844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-04T16:13:07.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:06 smithi083 ceph-mon[64844]: Upgrade: osd.5 is safe to restart 2024-03-04T16:13:08.215 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:07 smithi086 ceph-mon[57919]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-04T16:13:08.215 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:08.215 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T16:13:08.215 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:08.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:07 smithi083 ceph-mon[67383]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-04T16:13:08.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:08.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T16:13:08.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:08.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:07 smithi083 ceph-mon[64844]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-04T16:13:08.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:08.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T16:13:08.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:09.144 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:08 smithi086 ceph-mon[57919]: Upgrade: Updating osd.5 2024-03-04T16:13:09.144 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:08 smithi086 ceph-mon[57919]: Deploying daemon osd.5 on smithi086 2024-03-04T16:13:09.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:08 smithi083 ceph-mon[67383]: Upgrade: Updating osd.5 2024-03-04T16:13:09.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:08 smithi083 ceph-mon[67383]: Deploying daemon osd.5 on smithi086 2024-03-04T16:13:09.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:08 smithi083 ceph-mon[64844]: Upgrade: Updating osd.5 2024-03-04T16:13:09.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:08 smithi083 ceph-mon[64844]: Deploying daemon osd.5 on smithi086 2024-03-04T16:13:10.201 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:09 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:10.201 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:09 smithi086 ceph-mon[57919]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 145 op/s 2024-03-04T16:13:10.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:09 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:10.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:09 smithi083 ceph-mon[67383]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 145 op/s 2024-03-04T16:13:10.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:09 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:10.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:09 smithi083 ceph-mon[64844]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 145 op/s 2024-03-04T16:13:10.973 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:10 smithi086 systemd[1]: Stopping Ceph osd.5 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:13:10.973 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:10 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[31802]: 2024-03-04T16:13:10.813+0000 7fe9468f4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:13:10.973 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:10 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[31802]: 2024-03-04T16:13:10.813+0000 7fe9468f4700 -1 osd.5 131 *** Got signal Terminated *** 2024-03-04T16:13:10.973 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:10 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[31802]: 2024-03-04T16:13:10.813+0000 7fe9468f4700 -1 osd.5 131 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:13:11.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:10 smithi086 ceph-mon[57919]: osd.5 marked itself down and dead 2024-03-04T16:13:11.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:10 smithi083 ceph-mon[64844]: osd.5 marked itself down and dead 2024-03-04T16:13:11.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:10 smithi083 ceph-mon[67383]: osd.5 marked itself down and dead 2024-03-04T16:13:11.624 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:11 smithi086 podman[63073]: 2024-03-04 16:13:11.332742617 +0000 UTC m=+0.603252401 container died b82ca42b73533a4fbec05c8c6c8cbeefbbb0bddeed2fb76c2eb824c0bbae86d8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5, ceph=True, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, release=754, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-04T16:13:11.974 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:11 smithi086 podman[63073]: 2024-03-04 16:13:11.705842151 +0000 UTC m=+0.976351935 container cleanup b82ca42b73533a4fbec05c8c6c8cbeefbbb0bddeed2fb76c2eb824c0bbae86d8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_BRANCH=HEAD, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, name=centos-stream, maintainer=Guillaume Abrioux , release=754) 2024-03-04T16:13:11.974 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:11 smithi086 bash[63073]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5 2024-03-04T16:13:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:11 smithi086 ceph-mon[57919]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-03-04T16:13:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:11 smithi086 ceph-mon[57919]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:13:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:11 smithi086 ceph-mon[57919]: osdmap e132: 8 total, 7 up, 8 in 2024-03-04T16:13:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:12.234 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:12 smithi086 podman[63100]: 2024-03-04 16:13:12.074522585 +0000 UTC m=+0.741575533 container remove b82ca42b73533a4fbec05c8c6c8cbeefbbb0bddeed2fb76c2eb824c0bbae86d8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, RELEASE=HEAD, distribution-scope=public, GIT_CLEAN=True, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, io.k8s.display-name=CentOS Stream 8, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, version=8, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git) 2024-03-04T16:13:12.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:11 smithi083 ceph-mon[64844]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-03-04T16:13:12.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:11 smithi083 ceph-mon[64844]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:13:12.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:11 smithi083 ceph-mon[64844]: osdmap e132: 8 total, 7 up, 8 in 2024-03-04T16:13:12.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:12.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:12.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:11 smithi083 ceph-mon[67383]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-03-04T16:13:12.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:11 smithi083 ceph-mon[67383]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:13:12.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:11 smithi083 ceph-mon[67383]: osdmap e132: 8 total, 7 up, 8 in 2024-03-04T16:13:12.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:12.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:12.734 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:12 smithi086 podman[63192]: 2024-03-04 16:13:12.366761655 +0000 UTC m=+0.017825597 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:13:12.734 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:12 smithi086 podman[63192]: 2024-03-04 16:13:12.550707639 +0000 UTC m=+0.201771611 container create 9f1b5cc3409ebcc944cc2299f6542ada0a4245b49b5d520fdb86c3a1d8e87308 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:13:13.077 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:12 smithi086 ceph-mon[57919]: osdmap e133: 8 total, 7 up, 8 in 2024-03-04T16:13:13.077 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:12 smithi086 podman[63192]: 2024-03-04 16:13:12.786473462 +0000 UTC m=+0.437537393 container init 9f1b5cc3409ebcc944cc2299f6542ada0a4245b49b5d520fdb86c3a1d8e87308 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:13:13.077 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:12 smithi086 podman[63192]: 2024-03-04 16:13:12.789598607 +0000 UTC m=+0.440662546 container start 9f1b5cc3409ebcc944cc2299f6542ada0a4245b49b5d520fdb86c3a1d8e87308 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-04T16:13:13.077 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:12 smithi086 podman[63192]: 2024-03-04 16:13:12.886468109 +0000 UTC m=+0.537532040 container attach 9f1b5cc3409ebcc944cc2299f6542ada0a4245b49b5d520fdb86c3a1d8e87308 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=) 2024-03-04T16:13:13.372 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:13 smithi086 podman[63192]: 2024-03-04 16:13:13.077748858 +0000 UTC m=+0.728812790 container died 9f1b5cc3409ebcc944cc2299f6542ada0a4245b49b5d520fdb86c3a1d8e87308 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) 2024-03-04T16:13:13.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:12 smithi083 ceph-mon[64844]: osdmap e133: 8 total, 7 up, 8 in 2024-03-04T16:13:13.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:12 smithi083 ceph-mon[67383]: osdmap e133: 8 total, 7 up, 8 in 2024-03-04T16:13:13.984 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:13 smithi086 podman[63218]: 2024-03-04 16:13:13.648144832 +0000 UTC m=+0.563667657 container remove 9f1b5cc3409ebcc944cc2299f6542ada0a4245b49b5d520fdb86c3a1d8e87308 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-04T16:13:13.984 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:13 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.5.service: Deactivated successfully. 2024-03-04T16:13:13.984 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:13 smithi086 systemd[1]: Stopped Ceph osd.5 for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:13:13.984 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:13 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.5.service: Consumed 8.582s CPU time. 2024-03-04T16:13:13.985 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:13 smithi086 systemd[1]: Starting Ceph osd.5 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:13:14.252 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:13 smithi086 ceph-mon[57919]: pgmap v158: 161 pgs: 6 active+undersized+degraded, 12 peering, 14 stale+active+clean, 9 active+undersized, 120 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 0 B/s wr, 167 op/s; 17/630 objects degraded (2.698%) 2024-03-04T16:13:14.252 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:13 smithi086 ceph-mon[57919]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-04T16:13:14.252 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:13 smithi086 ceph-mon[57919]: Health check failed: Degraded data redundancy: 17/630 objects degraded (2.698%), 6 pgs degraded (PG_DEGRADED) 2024-03-04T16:13:14.252 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 podman[63325]: 2024-03-04 16:13:13.992927439 +0000 UTC m=+0.018969672 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:13:14.253 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 podman[63325]: 2024-03-04 16:13:14.091098366 +0000 UTC m=+0.117140588 container create 34a245bc07b2bcb942f56fe3d9950f600ab886b9b128cbdd3cb7481480ca25b8 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-04T16:13:14.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:13 smithi083 ceph-mon[64844]: pgmap v158: 161 pgs: 6 active+undersized+degraded, 12 peering, 14 stale+active+clean, 9 active+undersized, 120 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 0 B/s wr, 167 op/s; 17/630 objects degraded (2.698%) 2024-03-04T16:13:14.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:13 smithi083 ceph-mon[64844]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-04T16:13:14.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:13 smithi083 ceph-mon[64844]: Health check failed: Degraded data redundancy: 17/630 objects degraded (2.698%), 6 pgs degraded (PG_DEGRADED) 2024-03-04T16:13:14.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:13 smithi083 ceph-mon[67383]: pgmap v158: 161 pgs: 6 active+undersized+degraded, 12 peering, 14 stale+active+clean, 9 active+undersized, 120 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 0 B/s wr, 167 op/s; 17/630 objects degraded (2.698%) 2024-03-04T16:13:14.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:13 smithi083 ceph-mon[67383]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-04T16:13:14.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:13 smithi083 ceph-mon[67383]: Health check failed: Degraded data redundancy: 17/630 objects degraded (2.698%), 6 pgs degraded (PG_DEGRADED) 2024-03-04T16:13:14.734 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 podman[63325]: 2024-03-04 16:13:14.352053482 +0000 UTC m=+0.378095713 container init 34a245bc07b2bcb942f56fe3d9950f600ab886b9b128cbdd3cb7481480ca25b8 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:13:14.734 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 podman[63325]: 2024-03-04 16:13:14.356055013 +0000 UTC m=+0.382097240 container start 34a245bc07b2bcb942f56fe3d9950f600ab886b9b128cbdd3cb7481480ca25b8 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-activate, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-03-04T16:13:14.734 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 podman[63325]: 2024-03-04 16:13:14.42695089 +0000 UTC m=+0.452993127 container attach 34a245bc07b2bcb942f56fe3d9950f600ab886b9b128cbdd3cb7481480ca25b8 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS) 2024-03-04T16:13:15.155 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:15 smithi086 ceph-mon[57919]: pgmap v159: 161 pgs: 6 active+undersized+degraded, 12 peering, 14 stale+active+clean, 9 active+undersized, 120 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 0 B/s wr, 166 op/s; 17/630 objects degraded (2.698%) 2024-03-04T16:13:15.155 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-activate[63340]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-04T16:13:15.156 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-activate[63340]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-04T16:13:15.156 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 bash[63325]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-04T16:13:15.156 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 bash[63325]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-04T16:13:15.156 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 bash[63325]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-04T16:13:15.156 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 bash[63325]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-04T16:13:15.156 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 bash[63325]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-04T16:13:15.156 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 bash[63325]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-04T16:13:15.156 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 bash[63325]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-04T16:13:15.156 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-activate[63340]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-04T16:13:15.156 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-activate[63340]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-04T16:13:15.156 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-activate[63340]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-04T16:13:15.156 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-activate[63340]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-04T16:13:15.156 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-activate[63340]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-04T16:13:15.156 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:14 smithi086 podman[63325]: 2024-03-04 16:13:14.819109135 +0000 UTC m=+0.845151367 container died 34a245bc07b2bcb942f56fe3d9950f600ab886b9b128cbdd3cb7481480ca25b8 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-activate, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:13:15.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:14 smithi083 ceph-mon[64844]: pgmap v159: 161 pgs: 6 active+undersized+degraded, 12 peering, 14 stale+active+clean, 9 active+undersized, 120 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 0 B/s wr, 166 op/s; 17/630 objects degraded (2.698%) 2024-03-04T16:13:15.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:14 smithi083 ceph-mon[67383]: pgmap v159: 161 pgs: 6 active+undersized+degraded, 12 peering, 14 stale+active+clean, 9 active+undersized, 120 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 0 B/s wr, 166 op/s; 17/630 objects degraded (2.698%) 2024-03-04T16:13:15.734 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:15 smithi086 podman[63457]: 2024-03-04 16:13:15.430681641 +0000 UTC m=+0.606522087 container remove 34a245bc07b2bcb942f56fe3d9950f600ab886b9b128cbdd3cb7481480ca25b8 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-activate, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-04T16:13:15.734 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:15 smithi086 podman[63517]: 2024-03-04 16:13:15.547842796 +0000 UTC m=+0.018912636 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:13:15.734 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:15 smithi086 podman[63517]: 2024-03-04 16:13:15.665082484 +0000 UTC m=+0.136152312 container create 53215200f312ac0161d71dc5189f8f7ffdc06976d00fda29047f2ddd3268a093 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-04T16:13:15.987 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:13:15 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:13:15] "GET /metrics HTTP/1.1" 200 38514 "" "Prometheus/2.33.4" 2024-03-04T16:13:16.234 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:15 smithi086 podman[63517]: 2024-03-04 16:13:15.892698346 +0000 UTC m=+0.363768181 container init 53215200f312ac0161d71dc5189f8f7ffdc06976d00fda29047f2ddd3268a093 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-04T16:13:16.234 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:15 smithi086 podman[63517]: 2024-03-04 16:13:15.895721789 +0000 UTC m=+0.366791613 container start 53215200f312ac0161d71dc5189f8f7ffdc06976d00fda29047f2ddd3268a093 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:13:16.234 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:15 smithi086 ceph-osd[63536]: -- 172.21.15.86:0/2757258928 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560b25f17680 con 0x560b25ee0c00 2024-03-04T16:13:16.234 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:15 smithi086 bash[63517]: 53215200f312ac0161d71dc5189f8f7ffdc06976d00fda29047f2ddd3268a093 2024-03-04T16:13:16.235 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:15 smithi086 systemd[1]: Started Ceph osd.5 for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:13:16.734 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:16 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:13:16.442+0000 7f71d223e7c0 -1 Falling back to public interface 2024-03-04T16:13:17.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:17 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:17.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:17 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:17.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:17 smithi086 ceph-mon[57919]: pgmap v160: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 138 op/s; 70/630 objects degraded (11.111%) 2024-03-04T16:13:17.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:17 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:17.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:17 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:17.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:17 smithi083 ceph-mon[64844]: pgmap v160: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 138 op/s; 70/630 objects degraded (11.111%) 2024-03-04T16:13:17.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:17 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:17.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:17 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:17.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:17 smithi083 ceph-mon[67383]: pgmap v160: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 138 op/s; 70/630 objects degraded (11.111%) 2024-03-04T16:13:18.397 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:18 smithi086 ceph-mon[57919]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-04T16:13:18.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:18 smithi083 ceph-mon[64844]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-04T16:13:18.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:18 smithi083 ceph-mon[67383]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-04T16:13:19.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:19 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:19.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:19 smithi086 ceph-mon[57919]: pgmap v161: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 118 op/s; 70/630 objects degraded (11.111%) 2024-03-04T16:13:19.235 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:19 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:13:19.148+0000 7f71d223e7c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-03-04T16:13:19.236 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:19 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:13:19.175+0000 7f71d223e7c0 -1 osd.5 131 log_to_monitors true 2024-03-04T16:13:19.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:19 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:19.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:19 smithi083 ceph-mon[64844]: pgmap v161: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 118 op/s; 70/630 objects degraded (11.111%) 2024-03-04T16:13:19.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:19 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:19.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:19 smithi083 ceph-mon[67383]: pgmap v161: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 118 op/s; 70/630 objects degraded (11.111%) 2024-03-04T16:13:20.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:20 smithi086 ceph-mon[57919]: from='osd.5 [v2:172.21.15.86:6808/2336671410,v1:172.21.15.86:6809/2336671410]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T16:13:20.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:20 smithi086 ceph-mon[57919]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T16:13:20.487 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:20 smithi083 ceph-mon[67383]: from='osd.5 [v2:172.21.15.86:6808/2336671410,v1:172.21.15.86:6809/2336671410]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T16:13:20.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:20 smithi083 ceph-mon[67383]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T16:13:20.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:20 smithi083 ceph-mon[64844]: from='osd.5 [v2:172.21.15.86:6808/2336671410,v1:172.21.15.86:6809/2336671410]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T16:13:20.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:20 smithi083 ceph-mon[64844]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T16:13:20.984 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:13:20 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:13:20.507+0000 7f71bf68e700 -1 osd.5 131 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:13:21.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:21 smithi086 ceph-mon[57919]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-04T16:13:21.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:21 smithi086 ceph-mon[57919]: osdmap e134: 8 total, 7 up, 8 in 2024-03-04T16:13:21.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:21 smithi086 ceph-mon[57919]: from='osd.5 [v2:172.21.15.86:6808/2336671410,v1:172.21.15.86:6809/2336671410]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:13:21.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:21 smithi086 ceph-mon[57919]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:13:21.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:21 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:21.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:21 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:21.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:21 smithi086 ceph-mon[57919]: from='osd.5 ' entity='osd.5' 2024-03-04T16:13:21.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:21 smithi086 ceph-mon[57919]: pgmap v163: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 31 op/s; 70/630 objects degraded (11.111%) 2024-03-04T16:13:21.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:21 smithi083 ceph-mon[67383]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-04T16:13:21.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:21 smithi083 ceph-mon[67383]: osdmap e134: 8 total, 7 up, 8 in 2024-03-04T16:13:21.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:21 smithi083 ceph-mon[67383]: from='osd.5 [v2:172.21.15.86:6808/2336671410,v1:172.21.15.86:6809/2336671410]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:13:21.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:21 smithi083 ceph-mon[67383]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:13:21.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:21 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:21.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:21 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:21.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:21 smithi083 ceph-mon[67383]: from='osd.5 ' entity='osd.5' 2024-03-04T16:13:21.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:21 smithi083 ceph-mon[67383]: pgmap v163: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 31 op/s; 70/630 objects degraded (11.111%) 2024-03-04T16:13:21.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:21 smithi083 ceph-mon[64844]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-04T16:13:21.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:21 smithi083 ceph-mon[64844]: osdmap e134: 8 total, 7 up, 8 in 2024-03-04T16:13:21.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:21 smithi083 ceph-mon[64844]: from='osd.5 [v2:172.21.15.86:6808/2336671410,v1:172.21.15.86:6809/2336671410]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:13:21.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:21 smithi083 ceph-mon[64844]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:13:21.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:21 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:21.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:21 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:21.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:21 smithi083 ceph-mon[64844]: from='osd.5 ' entity='osd.5' 2024-03-04T16:13:21.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:21 smithi083 ceph-mon[64844]: pgmap v163: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 31 op/s; 70/630 objects degraded (11.111%) 2024-03-04T16:13:22.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:22 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:22.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:22 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:22.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:22 smithi086 ceph-mon[57919]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:13:22.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:22 smithi086 ceph-mon[57919]: osd.5 [v2:172.21.15.86:6808/2336671410,v1:172.21.15.86:6809/2336671410] boot 2024-03-04T16:13:22.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:22 smithi086 ceph-mon[57919]: osdmap e135: 8 total, 8 up, 8 in 2024-03-04T16:13:22.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:22 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:13:22.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:22 smithi086 ceph-mon[57919]: Health check update: Degraded data redundancy: 70/630 objects degraded (11.111%), 18 pgs degraded (PG_DEGRADED) 2024-03-04T16:13:22.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:22 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:22.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:22 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:22.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:22 smithi083 ceph-mon[64844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:13:22.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:22 smithi083 ceph-mon[64844]: osd.5 [v2:172.21.15.86:6808/2336671410,v1:172.21.15.86:6809/2336671410] boot 2024-03-04T16:13:22.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:22 smithi083 ceph-mon[64844]: osdmap e135: 8 total, 8 up, 8 in 2024-03-04T16:13:22.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:22 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:13:22.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:22 smithi083 ceph-mon[64844]: Health check update: Degraded data redundancy: 70/630 objects degraded (11.111%), 18 pgs degraded (PG_DEGRADED) 2024-03-04T16:13:22.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:22 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:22.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:22 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:22.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:22 smithi083 ceph-mon[67383]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:13:22.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:22 smithi083 ceph-mon[67383]: osd.5 [v2:172.21.15.86:6808/2336671410,v1:172.21.15.86:6809/2336671410] boot 2024-03-04T16:13:22.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:22 smithi083 ceph-mon[67383]: osdmap e135: 8 total, 8 up, 8 in 2024-03-04T16:13:22.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:22 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:13:22.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:22 smithi083 ceph-mon[67383]: Health check update: Degraded data redundancy: 70/630 objects degraded (11.111%), 18 pgs degraded (PG_DEGRADED) 2024-03-04T16:13:23.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:23 smithi086 ceph-mon[57919]: osdmap e136: 8 total, 8 up, 8 in 2024-03-04T16:13:23.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:23 smithi086 ceph-mon[57919]: pgmap v166: 161 pgs: 14 active+undersized+degraded, 34 active+undersized, 113 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 58/630 objects degraded (9.206%) 2024-03-04T16:13:23.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:23 smithi083 ceph-mon[64844]: osdmap e136: 8 total, 8 up, 8 in 2024-03-04T16:13:23.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:23 smithi083 ceph-mon[64844]: pgmap v166: 161 pgs: 14 active+undersized+degraded, 34 active+undersized, 113 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 58/630 objects degraded (9.206%) 2024-03-04T16:13:23.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:23 smithi083 ceph-mon[67383]: osdmap e136: 8 total, 8 up, 8 in 2024-03-04T16:13:23.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:23 smithi083 ceph-mon[67383]: pgmap v166: 161 pgs: 14 active+undersized+degraded, 34 active+undersized, 113 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 58/630 objects degraded (9.206%) 2024-03-04T16:13:24.348 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:13:25.008 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:13:25.008 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (6m) 73s ago 11m 24.9M - 0.23.0 ba2b418f427c 3c6dc3779a0e 2024-03-04T16:13:25.008 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (5m) 4s ago 10m 54.1M - 8.3.5 dad864ee21e9 fb754d58c53f 2024-03-04T16:13:25.008 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 running (5m) 73s ago 10m 53.6M - 3.5 e1d6a67b021e d58e71ccf9db 2024-03-04T16:13:25.008 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443,9283,8765 running (4m) 4s ago 15m 451M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c33100cb8a8 2024-03-04T16:13:25.008 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:8443,9283,8765 running (5m) 73s ago 17m 506M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 164dd93609ee 2024-03-04T16:13:25.008 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (4m) 73s ago 17m 48.8M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c173a2442197 2024-03-04T16:13:25.008 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (3m) 4s ago 15m 39.5M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 71557985ed5a 2024-03-04T16:13:25.009 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (4m) 73s ago 16m 40.6M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e eff1982ba275 2024-03-04T16:13:25.009 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (6m) 73s ago 11m 17.9M - 1.3.1 1dbe0e931976 9bf54b460640 2024-03-04T16:13:25.009 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (5m) 4s ago 11m 15.6M - 1.3.1 1dbe0e931976 b72dd80b2588 2024-03-04T16:13:25.009 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (3m) 73s ago 15m 44.4M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e f7ecdc5b1520 2024-03-04T16:13:25.009 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (2m) 73s ago 15m 46.6M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 2c648d9a2b38 2024-03-04T16:13:25.009 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (113s) 73s ago 14m 53.0M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c971c1f3f0a 2024-03-04T16:13:25.009 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (78s) 73s ago 14m 11.6M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 8991d467d41c 2024-03-04T16:13:25.009 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (43s) 4s ago 13m 53.6M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 912383028c72 2024-03-04T16:13:25.009 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (9s) 4s ago 13m 11.6M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 53215200f312 2024-03-04T16:13:25.009 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (13m) 4s ago 13m 57.2M 2992M 17.2.0 e1d6a67b021e 18cedf14321d 2024-03-04T16:13:25.009 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (12m) 4s ago 12m 62.4M 2992M 17.2.0 e1d6a67b021e 390e1798c738 2024-03-04T16:13:25.009 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 running (4m) 4s ago 12m 71.5M - 2.33.4 514e6a882f6e b563a246eeac 2024-03-04T16:13:25.009 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (10m) 73s ago 10m 74.5M - 17.2.0 e1d6a67b021e 65f14042f354 2024-03-04T16:13:25.009 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (10m) 4s ago 10m 75.8M - 17.2.0 e1d6a67b021e 78175cc6dc5d 2024-03-04T16:13:25.010 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (10m) 73s ago 10m 72.6M - 17.2.0 e1d6a67b021e 01dcf858bd62 2024-03-04T16:13:25.010 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (10m) 4s ago 10m 77.9M - 17.2.0 e1d6a67b021e 8f70a5ce9f0d 2024-03-04T16:13:25.394 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:13:25.394 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:13:25.394 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 3 2024-03-04T16:13:25.394 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:13:25.394 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:13:25.394 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:13:25.394 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:13:25.394 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:13:25.394 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-04T16:13:25.395 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 6 2024-03-04T16:13:25.395 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:13:25.395 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:13:25.395 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-04T16:13:25.395 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:13:25.395 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:13:25.395 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-04T16:13:25.395 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 11 2024-03-04T16:13:25.395 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:13:25.395 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:13:25.395 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:13:25.107+0000 7fe862dc1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-03-04T16:13:25.719 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:13:25.719 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:13:25.719 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:13:25.719 INFO:teuthology.orchestra.run.smithi083.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:13:25.720 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [ 2024-03-04T16:13:25.720 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr", 2024-03-04T16:13:25.720 INFO:teuthology.orchestra.run.smithi083.stdout: "mon" 2024-03-04T16:13:25.720 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-04T16:13:25.720 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "11/23 daemons upgraded", 2024-03-04T16:13:25.720 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "Currently upgrading osd daemons", 2024-03-04T16:13:25.720 INFO:teuthology.orchestra.run.smithi083.stdout: "is_paused": false 2024-03-04T16:13:25.720 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:13:25.987 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:13:25] "GET /metrics HTTP/1.1" 200 38514 "" "Prometheus/2.33.4" 2024-03-04T16:13:25.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[64844]: from='client.44457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:25.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[64844]: from='client.54394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:25.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[64844]: pgmap v167: 161 pgs: 14 active+undersized+degraded, 34 active+undersized, 113 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 58/630 objects degraded (9.206%) 2024-03-04T16:13:25.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:25.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:25.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:25.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:13:25.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:25.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:13:25.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:25.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:25.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:25.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-04T16:13:25.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/2774224279' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:25.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[67383]: from='client.44457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:25.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[67383]: from='client.54394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:25.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[67383]: pgmap v167: 161 pgs: 14 active+undersized+degraded, 34 active+undersized, 113 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 58/630 objects degraded (9.206%) 2024-03-04T16:13:25.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:25.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:25.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:25.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:13:25.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:25.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:13:25.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:25.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:25.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:25.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-04T16:13:25.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:25 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/2774224279' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:26.104 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_WARN Degraded data redundancy: 58/630 objects degraded (9.206%), 14 pgs degraded 2024-03-04T16:13:26.104 INFO:teuthology.orchestra.run.smithi083.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 58/630 objects degraded (9.206%), 14 pgs degraded 2024-03-04T16:13:26.104 INFO:teuthology.orchestra.run.smithi083.stdout: pg 2.12 is active+undersized+degraded, acting [3,7] 2024-03-04T16:13:26.104 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-03-04T16:13:26.104 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2024-03-04T16:13:26.104 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-03-04T16:13:26.104 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2024-03-04T16:13:26.104 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-03-04T16:13:26.104 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-03-04T16:13:26.104 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2024-03-04T16:13:26.104 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2024-03-04T16:13:26.105 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-03-04T16:13:26.105 INFO:teuthology.orchestra.run.smithi083.stdout: pg 4.3 is active+undersized+degraded, acting [0,7] 2024-03-04T16:13:26.105 INFO:teuthology.orchestra.run.smithi083.stdout: pg 4.15 is active+undersized+degraded, acting [7,3] 2024-03-04T16:13:26.105 INFO:teuthology.orchestra.run.smithi083.stdout: pg 6.c is active+undersized+degraded, acting [3,6] 2024-03-04T16:13:26.105 INFO:teuthology.orchestra.run.smithi083.stdout: pg 6.1a is active+undersized+degraded, acting [4,1] 2024-03-04T16:13:26.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:25 smithi086 ceph-mon[57919]: from='client.44457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:26.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:25 smithi086 ceph-mon[57919]: from='client.54394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:26.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:25 smithi086 ceph-mon[57919]: pgmap v167: 161 pgs: 14 active+undersized+degraded, 34 active+undersized, 113 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 58/630 objects degraded (9.206%) 2024-03-04T16:13:26.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:25 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:26.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:25 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:26.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:25 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:26.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:25 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:13:26.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:25 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:26.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:25 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:13:26.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:25 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:26.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:25 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:26.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:25 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:26.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:25 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-04T16:13:26.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:25 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/2774224279' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:27.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:26 smithi086 ceph-mon[57919]: from='client.44469 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:27.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:26 smithi086 ceph-mon[57919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-04T16:13:27.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:26 smithi086 ceph-mon[57919]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-03-04T16:13:27.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:26 smithi086 ceph-mon[57919]: from='client.44481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:27.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:26 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/1094902487' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:13:27.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:26 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:27.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:26 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:27.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:26 smithi086 ceph-mon[57919]: Health check update: Degraded data redundancy: 58/630 objects degraded (9.206%), 14 pgs degraded (PG_DEGRADED) 2024-03-04T16:13:27.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:26 smithi083 ceph-mon[67383]: from='client.44469 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:27.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:26 smithi083 ceph-mon[67383]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-04T16:13:27.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:26 smithi083 ceph-mon[67383]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-03-04T16:13:27.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:26 smithi083 ceph-mon[67383]: from='client.44481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:27.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:26 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/1094902487' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:13:27.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:26 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:27.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:26 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:27.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:26 smithi083 ceph-mon[67383]: Health check update: Degraded data redundancy: 58/630 objects degraded (9.206%), 14 pgs degraded (PG_DEGRADED) 2024-03-04T16:13:27.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:26 smithi083 ceph-mon[64844]: from='client.44469 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:27.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:26 smithi083 ceph-mon[64844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-04T16:13:27.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:26 smithi083 ceph-mon[64844]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-03-04T16:13:27.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:26 smithi083 ceph-mon[64844]: from='client.44481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:27.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:26 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/1094902487' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:13:27.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:26 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:27.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:26 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:27.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:26 smithi083 ceph-mon[64844]: Health check update: Degraded data redundancy: 58/630 objects degraded (9.206%), 14 pgs degraded (PG_DEGRADED) 2024-03-04T16:13:28.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:27 smithi086 ceph-mon[57919]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-04T16:13:28.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:27 smithi086 ceph-mon[57919]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 58/630 objects degraded (9.206%), 14 pgs degraded) 2024-03-04T16:13:28.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:27 smithi086 ceph-mon[57919]: Cluster is now healthy 2024-03-04T16:13:28.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:27 smithi083 ceph-mon[67383]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-04T16:13:28.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:27 smithi083 ceph-mon[67383]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 58/630 objects degraded (9.206%), 14 pgs degraded) 2024-03-04T16:13:28.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:27 smithi083 ceph-mon[67383]: Cluster is now healthy 2024-03-04T16:13:28.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:27 smithi083 ceph-mon[64844]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-04T16:13:28.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:27 smithi083 ceph-mon[64844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 58/630 objects degraded (9.206%), 14 pgs degraded) 2024-03-04T16:13:28.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:27 smithi083 ceph-mon[64844]: Cluster is now healthy 2024-03-04T16:13:30.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:29 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:30.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:29 smithi086 ceph-mon[57919]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:13:30.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:29 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:30.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:29 smithi083 ceph-mon[64844]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:13:30.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:29 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:30.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:29 smithi083 ceph-mon[67383]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:13:32.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:31 smithi086 ceph-mon[57919]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 548 B/s rd, 0 op/s 2024-03-04T16:13:32.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:31 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:32.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:31 smithi083 ceph-mon[64844]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 548 B/s rd, 0 op/s 2024-03-04T16:13:32.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:31 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:32.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:31 smithi083 ceph-mon[67383]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 548 B/s rd, 0 op/s 2024-03-04T16:13:32.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:31 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:34.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:33 smithi086 ceph-mon[57919]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 990 B/s rd, 0 op/s 2024-03-04T16:13:34.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:33 smithi083 ceph-mon[64844]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 990 B/s rd, 0 op/s 2024-03-04T16:13:34.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:33 smithi083 ceph-mon[67383]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 990 B/s rd, 0 op/s 2024-03-04T16:13:35.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:35 smithi083 ceph-mon[67383]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:13:35.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:35 smithi083 ceph-mon[64844]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:13:35.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:13:35 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:13:35] "GET /metrics HTTP/1.1" 200 38525 "" "Prometheus/2.33.4" 2024-03-04T16:13:36.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:35 smithi086 ceph-mon[57919]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:13:38.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:37 smithi086 ceph-mon[57919]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:13:38.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:37 smithi083 ceph-mon[64844]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:13:38.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:37 smithi083 ceph-mon[67383]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:13:40.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:39 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:40.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:39 smithi086 ceph-mon[57919]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:13:40.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:39 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:40.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:39 smithi083 ceph-mon[64844]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:13:40.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:39 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:40.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:39 smithi083 ceph-mon[67383]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:13:41.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:40 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-04T16:13:41.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:40 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-04T16:13:41.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:40 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-04T16:13:42.349 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:41 smithi086 ceph-mon[57919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-04T16:13:42.349 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:41 smithi086 ceph-mon[57919]: Upgrade: osd.6 is safe to restart 2024-03-04T16:13:42.349 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:41 smithi086 ceph-mon[57919]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:13:42.349 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:42.349 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-04T16:13:42.349 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:42.349 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:42.349 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:42.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:41 smithi083 ceph-mon[64844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-04T16:13:42.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:41 smithi083 ceph-mon[64844]: Upgrade: osd.6 is safe to restart 2024-03-04T16:13:42.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:41 smithi083 ceph-mon[64844]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:13:42.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:42.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-04T16:13:42.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:42.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:42.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:42.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:41 smithi083 ceph-mon[67383]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-04T16:13:42.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:41 smithi083 ceph-mon[67383]: Upgrade: osd.6 is safe to restart 2024-03-04T16:13:42.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:41 smithi083 ceph-mon[67383]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:13:42.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:42.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-04T16:13:42.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:42.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:42.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:43.279 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:42 smithi086 ceph-mon[57919]: Upgrade: Updating osd.6 2024-03-04T16:13:43.280 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:42 smithi086 ceph-mon[57919]: Deploying daemon osd.6 on smithi086 2024-03-04T16:13:43.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:42 smithi083 ceph-mon[64844]: Upgrade: Updating osd.6 2024-03-04T16:13:43.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:42 smithi083 ceph-mon[64844]: Deploying daemon osd.6 on smithi086 2024-03-04T16:13:43.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:42 smithi083 ceph-mon[67383]: Upgrade: Updating osd.6 2024-03-04T16:13:43.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:42 smithi083 ceph-mon[67383]: Deploying daemon osd.6 on smithi086 2024-03-04T16:13:44.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:43 smithi086 ceph-mon[57919]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:13:44.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:43 smithi083 ceph-mon[67383]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:13:44.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:43 smithi083 ceph-mon[64844]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:13:45.234 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:44 smithi086 systemd[1]: Stopping Ceph osd.6 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:13:45.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:45 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[34681]: 2024-03-04T16:13:45.275+0000 7fcf9c3c0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:13:45.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:45 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[34681]: 2024-03-04T16:13:45.275+0000 7fcf9c3c0700 -1 osd.6 136 *** Got signal Terminated *** 2024-03-04T16:13:45.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:45 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[34681]: 2024-03-04T16:13:45.275+0000 7fcf9c3c0700 -1 osd.6 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:13:45.987 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:13:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:13:45] "GET /metrics HTTP/1.1" 200 38522 "" "Prometheus/2.33.4" 2024-03-04T16:13:46.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:45 smithi086 ceph-mon[57919]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:13:46.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:45 smithi086 ceph-mon[57919]: osd.6 marked itself down and dead 2024-03-04T16:13:46.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:45 smithi083 ceph-mon[64844]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:13:46.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:45 smithi083 ceph-mon[64844]: osd.6 marked itself down and dead 2024-03-04T16:13:46.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:45 smithi083 ceph-mon[67383]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:13:46.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:45 smithi083 ceph-mon[67383]: osd.6 marked itself down and dead 2024-03-04T16:13:46.809 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:46 smithi086 podman[65418]: 2024-03-04 16:13:46.522110772 +0000 UTC m=+1.370029913 container died 18cedf14321dfa2c5fbe4f569165299d2cd96105726af8f8ae7f5fafa02a5872 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, ceph=True, GIT_CLEAN=True, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, distribution-scope=public, vendor=Red Hat, Inc., io.buildah.version=1.19.8, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, name=centos-stream, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD) 2024-03-04T16:13:47.168 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:46 smithi086 ceph-mon[57919]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:13:47.168 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:46 smithi086 ceph-mon[57919]: osdmap e137: 8 total, 7 up, 8 in 2024-03-04T16:13:47.168 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:46 smithi086 ceph-mon[57919]: pgmap v179: 161 pgs: 3 peering, 11 stale+active+clean, 147 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-04T16:13:47.168 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:46 smithi086 podman[65418]: 2024-03-04 16:13:46.908589704 +0000 UTC m=+1.756508860 container cleanup 18cedf14321dfa2c5fbe4f569165299d2cd96105726af8f8ae7f5fafa02a5872 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6, maintainer=Guillaume Abrioux , release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, vendor=Red Hat, Inc., version=8, io.buildah.version=1.19.8, GIT_BRANCH=HEAD) 2024-03-04T16:13:47.169 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:46 smithi086 bash[65418]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6 2024-03-04T16:13:47.484 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:47 smithi086 podman[65436]: 2024-03-04 16:13:47.168434483 +0000 UTC m=+0.642824600 container remove 18cedf14321dfa2c5fbe4f569165299d2cd96105726af8f8ae7f5fafa02a5872 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6, io.openshift.expose-services=, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, architecture=x86_64, io.openshift.tags=base centos centos-stream, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., RELEASE=HEAD, name=centos-stream, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8) 2024-03-04T16:13:47.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:46 smithi083 ceph-mon[64844]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:13:47.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:46 smithi083 ceph-mon[64844]: osdmap e137: 8 total, 7 up, 8 in 2024-03-04T16:13:47.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:46 smithi083 ceph-mon[64844]: pgmap v179: 161 pgs: 3 peering, 11 stale+active+clean, 147 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-04T16:13:47.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:46 smithi083 ceph-mon[67383]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:13:47.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:46 smithi083 ceph-mon[67383]: osdmap e137: 8 total, 7 up, 8 in 2024-03-04T16:13:47.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:46 smithi083 ceph-mon[67383]: pgmap v179: 161 pgs: 3 peering, 11 stale+active+clean, 147 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-04T16:13:47.854 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:47 smithi086 podman[65531]: 2024-03-04 16:13:47.432800084 +0000 UTC m=+0.018933205 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:13:47.855 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:47 smithi086 podman[65531]: 2024-03-04 16:13:47.53594099 +0000 UTC m=+0.122074101 container create 715e3621d52e389d40ea5cef8856b80129e29e6fcd8e6d956b6dbc8b8287de9d (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-04T16:13:47.855 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:47 smithi086 podman[65531]: 2024-03-04 16:13:47.767452486 +0000 UTC m=+0.353585610 container init 715e3621d52e389d40ea5cef8856b80129e29e6fcd8e6d956b6dbc8b8287de9d (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:13:47.855 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:47 smithi086 podman[65531]: 2024-03-04 16:13:47.770983329 +0000 UTC m=+0.357116450 container start 715e3621d52e389d40ea5cef8856b80129e29e6fcd8e6d956b6dbc8b8287de9d (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=) 2024-03-04T16:13:47.855 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:47 smithi086 podman[65531]: 2024-03-04 16:13:47.854956762 +0000 UTC m=+0.441089878 container attach 715e3621d52e389d40ea5cef8856b80129e29e6fcd8e6d956b6dbc8b8287de9d (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:13:48.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:48 smithi086 ceph-mon[57919]: osdmap e138: 8 total, 7 up, 8 in 2024-03-04T16:13:48.235 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:48 smithi086 podman[65531]: 2024-03-04 16:13:48.053953732 +0000 UTC m=+0.640086853 container died 715e3621d52e389d40ea5cef8856b80129e29e6fcd8e6d956b6dbc8b8287de9d (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:13:48.487 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:48 smithi083 ceph-mon[67383]: osdmap e138: 8 total, 7 up, 8 in 2024-03-04T16:13:48.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:48 smithi083 ceph-mon[64844]: osdmap e138: 8 total, 7 up, 8 in 2024-03-04T16:13:48.987 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:48 smithi086 podman[65556]: 2024-03-04 16:13:48.650557487 +0000 UTC m=+0.591666637 container remove 715e3621d52e389d40ea5cef8856b80129e29e6fcd8e6d956b6dbc8b8287de9d (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.schema-version=1.0) 2024-03-04T16:13:48.987 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:48 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-04T16:13:48.988 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:48 smithi086 systemd[1]: Stopped Ceph osd.6 for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:13:48.988 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:48 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.6.service: Consumed 8.214s CPU time. 2024-03-04T16:13:48.988 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:48 smithi086 systemd[1]: Starting Ceph osd.6 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:13:49.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:49 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:49.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:49 smithi086 ceph-mon[57919]: pgmap v181: 161 pgs: 4 active+undersized+degraded, 3 peering, 10 stale+active+clean, 6 active+undersized, 138 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 16/630 objects degraded (2.540%) 2024-03-04T16:13:49.371 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 podman[65664]: 2024-03-04 16:13:48.990279309 +0000 UTC m=+0.019402657 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:13:49.372 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 podman[65664]: 2024-03-04 16:13:49.093340009 +0000 UTC m=+0.122463342 container create 724d09aa148765c3a85f5a5be7de2e42c5529d1dcbc027397cb883d9d26f6e0e (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux ) 2024-03-04T16:13:49.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:49 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:49.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:49 smithi083 ceph-mon[64844]: pgmap v181: 161 pgs: 4 active+undersized+degraded, 3 peering, 10 stale+active+clean, 6 active+undersized, 138 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 16/630 objects degraded (2.540%) 2024-03-04T16:13:49.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:49 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:49.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:49 smithi083 ceph-mon[67383]: pgmap v181: 161 pgs: 4 active+undersized+degraded, 3 peering, 10 stale+active+clean, 6 active+undersized, 138 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 16/630 objects degraded (2.540%) 2024-03-04T16:13:49.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 podman[65664]: 2024-03-04 16:13:49.371395556 +0000 UTC m=+0.400518894 container init 724d09aa148765c3a85f5a5be7de2e42c5529d1dcbc027397cb883d9d26f6e0e (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-activate, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-04T16:13:49.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 podman[65664]: 2024-03-04 16:13:49.37480804 +0000 UTC m=+0.403931381 container start 724d09aa148765c3a85f5a5be7de2e42c5529d1dcbc027397cb883d9d26f6e0e (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-activate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-04T16:13:49.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 podman[65664]: 2024-03-04 16:13:49.45422014 +0000 UTC m=+0.483343487 container attach 724d09aa148765c3a85f5a5be7de2e42c5529d1dcbc027397cb883d9d26f6e0e (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-activate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:13:50.125 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:50 smithi086 ceph-mon[57919]: Health check failed: Degraded data redundancy: 16/630 objects degraded (2.540%), 4 pgs degraded (PG_DEGRADED) 2024-03-04T16:13:50.125 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-activate[65680]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-04T16:13:50.125 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-activate[65680]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-04T16:13:50.125 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-activate[65680]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-04T16:13:50.125 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-activate[65680]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-04T16:13:50.125 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-activate[65680]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-04T16:13:50.126 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-activate[65680]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-04T16:13:50.126 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 bash[65664]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-04T16:13:50.126 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 bash[65664]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-04T16:13:50.126 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 bash[65664]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-04T16:13:50.126 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 bash[65664]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-04T16:13:50.126 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 bash[65664]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-04T16:13:50.126 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 bash[65664]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-04T16:13:50.126 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 bash[65664]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-04T16:13:50.126 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-activate[65680]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-04T16:13:50.126 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:49 smithi086 podman[65664]: 2024-03-04 16:13:49.837457168 +0000 UTC m=+0.866580500 container died 724d09aa148765c3a85f5a5be7de2e42c5529d1dcbc027397cb883d9d26f6e0e (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-activate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222) 2024-03-04T16:13:50.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:50 smithi083 ceph-mon[64844]: Health check failed: Degraded data redundancy: 16/630 objects degraded (2.540%), 4 pgs degraded (PG_DEGRADED) 2024-03-04T16:13:50.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:50 smithi083 ceph-mon[67383]: Health check failed: Degraded data redundancy: 16/630 objects degraded (2.540%), 4 pgs degraded (PG_DEGRADED) 2024-03-04T16:13:50.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:50 smithi086 podman[65813]: 2024-03-04 16:13:50.4246794 +0000 UTC m=+0.582516852 container remove 724d09aa148765c3a85f5a5be7de2e42c5529d1dcbc027397cb883d9d26f6e0e (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-activate, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:13:50.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:50 smithi086 podman[65881]: 2024-03-04 16:13:50.633968001 +0000 UTC m=+0.111416130 container create a4584e5b304436cf1961578cb306acc172287f9756df7e374440cac43300d828 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:13:50.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:50 smithi086 podman[65881]: 2024-03-04 16:13:50.540446305 +0000 UTC m=+0.017894399 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:13:51.027 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:50 smithi086 podman[65881]: 2024-03-04 16:13:50.861250125 +0000 UTC m=+0.338698223 container init a4584e5b304436cf1961578cb306acc172287f9756df7e374440cac43300d828 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:13:51.028 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:50 smithi086 podman[65881]: 2024-03-04 16:13:50.864258316 +0000 UTC m=+0.341706404 container start a4584e5b304436cf1961578cb306acc172287f9756df7e374440cac43300d828 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:13:51.028 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:50 smithi086 ceph-osd[65900]: -- 172.21.15.86:0/1833433079 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5619290ef680 con 0x5619290b9400 2024-03-04T16:13:51.028 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:50 smithi086 bash[65881]: a4584e5b304436cf1961578cb306acc172287f9756df7e374440cac43300d828 2024-03-04T16:13:51.028 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:50 smithi086 systemd[1]: Started Ceph osd.6 for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:13:51.409 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:51 smithi086 ceph-mon[57919]: pgmap v182: 161 pgs: 11 active+undersized+degraded, 3 peering, 2 stale+active+clean, 23 active+undersized, 122 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 50/630 objects degraded (7.937%) 2024-03-04T16:13:51.410 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:51 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:51.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:51 smithi083 ceph-mon[67383]: pgmap v182: 161 pgs: 11 active+undersized+degraded, 3 peering, 2 stale+active+clean, 23 active+undersized, 122 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 50/630 objects degraded (7.937%) 2024-03-04T16:13:51.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:51 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:51.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:51 smithi083 ceph-mon[64844]: pgmap v182: 161 pgs: 11 active+undersized+degraded, 3 peering, 2 stale+active+clean, 23 active+undersized, 122 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 50/630 objects degraded (7.937%) 2024-03-04T16:13:51.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:51 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:51.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:51 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:13:51.409+0000 7f0e764567c0 -1 Falling back to public interface 2024-03-04T16:13:52.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:52 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:52.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:52 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:52.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:52 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:53.419 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:53 smithi086 ceph-mon[57919]: pgmap v183: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 74/630 objects degraded (11.746%) 2024-03-04T16:13:53.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:53 smithi083 ceph-mon[64844]: pgmap v183: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 74/630 objects degraded (11.746%) 2024-03-04T16:13:53.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:53 smithi083 ceph-mon[67383]: pgmap v183: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 74/630 objects degraded (11.746%) 2024-03-04T16:13:54.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:54 smithi086 ceph-mon[57919]: from='osd.6 [v2:172.21.15.86:6816/2392320071,v1:172.21.15.86:6817/2392320071]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T16:13:54.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:54 smithi086 ceph-mon[57919]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T16:13:54.484 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:54 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:13:54.125+0000 7f0e764567c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-03-04T16:13:54.484 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:54 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:13:54.154+0000 7f0e764567c0 -1 osd.6 136 log_to_monitors true 2024-03-04T16:13:54.484 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:13:54 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:13:54.235+0000 7f0e678ae700 -1 osd.6 136 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:13:54.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:54 smithi083 ceph-mon[64844]: from='osd.6 [v2:172.21.15.86:6816/2392320071,v1:172.21.15.86:6817/2392320071]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T16:13:54.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:54 smithi083 ceph-mon[64844]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T16:13:54.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:54 smithi083 ceph-mon[67383]: from='osd.6 [v2:172.21.15.86:6816/2392320071,v1:172.21.15.86:6817/2392320071]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T16:13:54.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:54 smithi083 ceph-mon[67383]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T16:13:55.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:55 smithi083 ceph-mon[64844]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-04T16:13:55.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:55 smithi083 ceph-mon[64844]: osdmap e139: 8 total, 7 up, 8 in 2024-03-04T16:13:55.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:55 smithi083 ceph-mon[64844]: from='osd.6 [v2:172.21.15.86:6816/2392320071,v1:172.21.15.86:6817/2392320071]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:13:55.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:55 smithi083 ceph-mon[64844]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:13:55.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:55 smithi083 ceph-mon[64844]: pgmap v185: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 74/630 objects degraded (11.746%) 2024-03-04T16:13:55.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:55 smithi083 ceph-mon[67383]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-04T16:13:55.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:55 smithi083 ceph-mon[67383]: osdmap e139: 8 total, 7 up, 8 in 2024-03-04T16:13:55.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:55 smithi083 ceph-mon[67383]: from='osd.6 [v2:172.21.15.86:6816/2392320071,v1:172.21.15.86:6817/2392320071]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:13:55.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:55 smithi083 ceph-mon[67383]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:13:55.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:55 smithi083 ceph-mon[67383]: pgmap v185: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 74/630 objects degraded (11.746%) 2024-03-04T16:13:55.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:55 smithi086 ceph-mon[57919]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-04T16:13:55.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:55 smithi086 ceph-mon[57919]: osdmap e139: 8 total, 7 up, 8 in 2024-03-04T16:13:55.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:55 smithi086 ceph-mon[57919]: from='osd.6 [v2:172.21.15.86:6816/2392320071,v1:172.21.15.86:6817/2392320071]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:13:55.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:55 smithi086 ceph-mon[57919]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:13:55.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:55 smithi086 ceph-mon[57919]: pgmap v185: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 74/630 objects degraded (11.746%) 2024-03-04T16:13:55.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:13:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:13:55] "GET /metrics HTTP/1.1" 200 38522 "" "Prometheus/2.33.4" 2024-03-04T16:13:56.473 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:13:56.635 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:56 smithi086 ceph-mon[57919]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:13:56.636 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:56 smithi086 ceph-mon[57919]: osd.6 [v2:172.21.15.86:6816/2392320071,v1:172.21.15.86:6817/2392320071] boot 2024-03-04T16:13:56.636 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:56 smithi086 ceph-mon[57919]: osdmap e140: 8 total, 8 up, 8 in 2024-03-04T16:13:56.636 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:56 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:13:56.636 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:56 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:56.636 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:56 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:56.636 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:56 smithi086 ceph-mon[57919]: osdmap e141: 8 total, 8 up, 8 in 2024-03-04T16:13:56.737 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:56 smithi083 ceph-mon[64844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:13:56.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:56 smithi083 ceph-mon[64844]: osd.6 [v2:172.21.15.86:6816/2392320071,v1:172.21.15.86:6817/2392320071] boot 2024-03-04T16:13:56.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:56 smithi083 ceph-mon[64844]: osdmap e140: 8 total, 8 up, 8 in 2024-03-04T16:13:56.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:13:56.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:56.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:56.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:56 smithi083 ceph-mon[64844]: osdmap e141: 8 total, 8 up, 8 in 2024-03-04T16:13:56.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:56 smithi083 ceph-mon[67383]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:13:56.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:56 smithi083 ceph-mon[67383]: osd.6 [v2:172.21.15.86:6816/2392320071,v1:172.21.15.86:6817/2392320071] boot 2024-03-04T16:13:56.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:56 smithi083 ceph-mon[67383]: osdmap e140: 8 total, 8 up, 8 in 2024-03-04T16:13:56.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:56 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:13:56.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:56 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:56.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:56 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:56.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:56 smithi083 ceph-mon[67383]: osdmap e141: 8 total, 8 up, 8 in 2024-03-04T16:13:57.141 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:13:57.142 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (6m) 105s ago 11m 24.9M - 0.23.0 ba2b418f427c 3c6dc3779a0e 2024-03-04T16:13:57.142 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (6m) 1s ago 11m 54.1M - 8.3.5 dad864ee21e9 fb754d58c53f 2024-03-04T16:13:57.142 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 running (5m) 105s ago 10m 53.6M - 3.5 e1d6a67b021e d58e71ccf9db 2024-03-04T16:13:57.142 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443,9283,8765 running (5m) 1s ago 16m 452M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c33100cb8a8 2024-03-04T16:13:57.142 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:8443,9283,8765 running (5m) 105s ago 17m 506M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 164dd93609ee 2024-03-04T16:13:57.142 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (5m) 105s ago 17m 48.8M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c173a2442197 2024-03-04T16:13:57.142 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (4m) 1s ago 16m 40.0M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 71557985ed5a 2024-03-04T16:13:57.142 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (4m) 105s ago 16m 40.6M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e eff1982ba275 2024-03-04T16:13:57.142 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (6m) 105s ago 12m 17.9M - 1.3.1 1dbe0e931976 9bf54b460640 2024-03-04T16:13:57.142 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (6m) 1s ago 12m 15.6M - 1.3.1 1dbe0e931976 b72dd80b2588 2024-03-04T16:13:57.142 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (3m) 105s ago 15m 44.4M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e f7ecdc5b1520 2024-03-04T16:13:57.142 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (3m) 105s ago 15m 46.6M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 2c648d9a2b38 2024-03-04T16:13:57.142 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (2m) 105s ago 15m 53.0M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c971c1f3f0a 2024-03-04T16:13:57.142 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (110s) 105s ago 14m 11.6M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 8991d467d41c 2024-03-04T16:13:57.143 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (75s) 1s ago 14m 55.2M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 912383028c72 2024-03-04T16:13:57.143 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (41s) 1s ago 14m 50.3M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 53215200f312 2024-03-04T16:13:57.143 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (6s) 1s ago 13m 13.5M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e a4584e5b3044 2024-03-04T16:13:57.143 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (13m) 1s ago 13m 63.5M 2992M 17.2.0 e1d6a67b021e 390e1798c738 2024-03-04T16:13:57.143 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 running (5m) 1s ago 12m 71.5M - 2.33.4 514e6a882f6e b563a246eeac 2024-03-04T16:13:57.143 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (11m) 105s ago 11m 74.5M - 17.2.0 e1d6a67b021e 65f14042f354 2024-03-04T16:13:57.143 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (10m) 1s ago 10m 76.0M - 17.2.0 e1d6a67b021e 78175cc6dc5d 2024-03-04T16:13:57.143 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (10m) 105s ago 10m 72.6M - 17.2.0 e1d6a67b021e 01dcf858bd62 2024-03-04T16:13:57.143 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (10m) 1s ago 10m 78.1M - 17.2.0 e1d6a67b021e 8f70a5ce9f0d 2024-03-04T16:13:57.533 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:13:57.533 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:13:57.533 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 3 2024-03-04T16:13:57.533 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:13:57.533 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:13:57.533 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:13:57.533 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:13:57.533 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:13:57.534 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-04T16:13:57.534 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 7 2024-03-04T16:13:57.534 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:13:57.534 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:13:57.534 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-04T16:13:57.534 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:13:57.534 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:13:57.534 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-04T16:13:57.534 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 12 2024-03-04T16:13:57.534 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:13:57.534 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:13:57.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:57 smithi086 ceph-mon[57919]: from='client.44499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:57.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:57.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:57.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:57 smithi086 ceph-mon[57919]: from='client.54436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:57.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:57 smithi086 ceph-mon[57919]: pgmap v188: 161 pgs: 15 peering, 13 active+undersized+degraded, 25 active+undersized, 108 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 53/630 objects degraded (8.413%) 2024-03-04T16:13:57.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:57.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:57.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:57 smithi086 ceph-mon[57919]: Health check update: Degraded data redundancy: 74/630 objects degraded (11.746%), 18 pgs degraded (PG_DEGRADED) 2024-03-04T16:13:57.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:57.862 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:13:57.862 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:13:57.862 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:13:57.862 INFO:teuthology.orchestra.run.smithi083.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:13:57.862 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [ 2024-03-04T16:13:57.862 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr", 2024-03-04T16:13:57.862 INFO:teuthology.orchestra.run.smithi083.stdout: "mon" 2024-03-04T16:13:57.862 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-04T16:13:57.862 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "12/23 daemons upgraded", 2024-03-04T16:13:57.862 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "Currently upgrading osd daemons", 2024-03-04T16:13:57.863 INFO:teuthology.orchestra.run.smithi083.stdout: "is_paused": false 2024-03-04T16:13:57.863 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:13:57.863 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:57 smithi083 ceph-mon[64844]: from='client.44499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:57.863 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:57.863 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:57.863 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:57 smithi083 ceph-mon[64844]: from='client.54436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:57.863 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:57 smithi083 ceph-mon[64844]: pgmap v188: 161 pgs: 15 peering, 13 active+undersized+degraded, 25 active+undersized, 108 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 53/630 objects degraded (8.413%) 2024-03-04T16:13:57.863 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:57.863 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:57.864 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:57 smithi083 ceph-mon[64844]: Health check update: Degraded data redundancy: 74/630 objects degraded (11.746%), 18 pgs degraded (PG_DEGRADED) 2024-03-04T16:13:57.864 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:57.864 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:57 smithi083 ceph-mon[67383]: from='client.44499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:57.864 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:57.864 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:57.864 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:57 smithi083 ceph-mon[67383]: from='client.54436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:57.864 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:57 smithi083 ceph-mon[67383]: pgmap v188: 161 pgs: 15 peering, 13 active+undersized+degraded, 25 active+undersized, 108 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 53/630 objects degraded (8.413%) 2024-03-04T16:13:57.864 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:57.864 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:57.865 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:57 smithi083 ceph-mon[67383]: Health check update: Degraded data redundancy: 74/630 objects degraded (11.746%), 18 pgs degraded (PG_DEGRADED) 2024-03-04T16:13:57.865 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:13:58.262 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_WARN Degraded data redundancy: 53/630 objects degraded (8.413%), 13 pgs degraded 2024-03-04T16:13:58.263 INFO:teuthology.orchestra.run.smithi083.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 53/630 objects degraded (8.413%), 13 pgs degraded 2024-03-04T16:13:58.263 INFO:teuthology.orchestra.run.smithi083.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-03-04T16:13:58.263 INFO:teuthology.orchestra.run.smithi083.stdout: pg 2.7 is active+undersized+degraded, acting [7,2] 2024-03-04T16:13:58.263 INFO:teuthology.orchestra.run.smithi083.stdout: pg 2.14 is active+undersized+degraded, acting [3,5] 2024-03-04T16:13:58.263 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-03-04T16:13:58.263 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-03-04T16:13:58.263 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-03-04T16:13:58.263 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-03-04T16:13:58.263 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-03-04T16:13:58.263 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2024-03-04T16:13:58.263 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-03-04T16:13:58.264 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-03-04T16:13:58.264 INFO:teuthology.orchestra.run.smithi083.stdout: pg 4.e is active+undersized+degraded, acting [4,1] 2024-03-04T16:13:58.264 INFO:teuthology.orchestra.run.smithi083.stdout: pg 4.1f is active+undersized+degraded, acting [5,1] 2024-03-04T16:13:58.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:58 smithi086 ceph-mon[57919]: from='client.44511 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:58.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:58 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/2616529690' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:58.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:58 smithi086 ceph-mon[57919]: from='client.54454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:58.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:58 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/986905978' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:13:58.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:58 smithi083 ceph-mon[64844]: from='client.44511 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:58.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:58 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/2616529690' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:58.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:58 smithi083 ceph-mon[64844]: from='client.54454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:58.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:58 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/986905978' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:13:58.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:58 smithi083 ceph-mon[67383]: from='client.44511 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:58.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:58 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/2616529690' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:13:58.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:58 smithi083 ceph-mon[67383]: from='client.54454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:13:58.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:58 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/986905978' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:13:59.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:59 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:59.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:13:59 smithi086 ceph-mon[57919]: pgmap v189: 161 pgs: 15 peering, 10 active+undersized+degraded, 19 active+undersized, 117 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 38/630 objects degraded (6.032%) 2024-03-04T16:13:59.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:59 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:59.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:13:59 smithi083 ceph-mon[64844]: pgmap v189: 161 pgs: 15 peering, 10 active+undersized+degraded, 19 active+undersized, 117 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 38/630 objects degraded (6.032%) 2024-03-04T16:13:59.966 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:59 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:59.966 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:13:59 smithi083 ceph-mon[67383]: pgmap v189: 161 pgs: 15 peering, 10 active+undersized+degraded, 19 active+undersized, 117 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 38/630 objects degraded (6.032%) 2024-03-04T16:14:00.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:13:59 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:13:59.959+0000 7fe862dc1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-04T16:14:01.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:00 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:01.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:00 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:01.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:00 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:01.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:00 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:14:01.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:00 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:01.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:00 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:14:01.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:00 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:01.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:00 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:01.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:00 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:01.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:00 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-04T16:14:01.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:00 smithi086 ceph-mon[57919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-04T16:14:01.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:00 smithi086 ceph-mon[57919]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-04T16:14:01.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:01.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:01.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:01.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:14:01.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:01.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:14:01.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:01.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:01.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:01.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-04T16:14:01.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[64844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-04T16:14:01.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[64844]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-04T16:14:01.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:01.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:01.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:01.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:14:01.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:01.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:14:01.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:01.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:01.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:01.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-04T16:14:01.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[67383]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-04T16:14:01.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:00 smithi083 ceph-mon[67383]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-04T16:14:02.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:01 smithi086 ceph-mon[57919]: pgmap v190: 161 pgs: 15 peering, 146 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-03-04T16:14:02.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:01 smithi083 ceph-mon[67383]: pgmap v190: 161 pgs: 15 peering, 146 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-03-04T16:14:02.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:01 smithi083 ceph-mon[64844]: pgmap v190: 161 pgs: 15 peering, 146 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-03-04T16:14:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:02 smithi086 ceph-mon[57919]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/630 objects degraded (6.032%), 10 pgs degraded) 2024-03-04T16:14:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:02 smithi086 ceph-mon[57919]: Cluster is now healthy 2024-03-04T16:14:03.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:02 smithi083 ceph-mon[67383]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/630 objects degraded (6.032%), 10 pgs degraded) 2024-03-04T16:14:03.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:02 smithi083 ceph-mon[67383]: Cluster is now healthy 2024-03-04T16:14:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:02 smithi083 ceph-mon[64844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/630 objects degraded (6.032%), 10 pgs degraded) 2024-03-04T16:14:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:02 smithi083 ceph-mon[64844]: Cluster is now healthy 2024-03-04T16:14:04.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:03 smithi086 ceph-mon[57919]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:14:04.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:03 smithi083 ceph-mon[67383]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:14:04.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:03 smithi083 ceph-mon[64844]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:14:05.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:05 smithi083 ceph-mon[67383]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-04T16:14:05.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:05 smithi083 ceph-mon[64844]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-04T16:14:05.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:14:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:14:05] "GET /metrics HTTP/1.1" 200 38381 "" "Prometheus/2.33.4" 2024-03-04T16:14:06.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:05 smithi086 ceph-mon[57919]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-04T16:14:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:07 smithi086 ceph-mon[57919]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 0 op/s 2024-03-04T16:14:08.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:08.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:07 smithi083 ceph-mon[67383]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 0 op/s 2024-03-04T16:14:08.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:08.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:07 smithi083 ceph-mon[64844]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 0 op/s 2024-03-04T16:14:08.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:10.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:09 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:10.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:09 smithi083 ceph-mon[67383]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:14:10.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:09 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:10.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:09 smithi083 ceph-mon[64844]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:14:10.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:09 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:10.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:09 smithi086 ceph-mon[57919]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:14:12.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:11 smithi083 ceph-mon[64844]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:14:12.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:12.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:12.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:11 smithi083 ceph-mon[67383]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:14:12.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:12.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:12.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:11 smithi086 ceph-mon[57919]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:14:12.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:12.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:14.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:13 smithi086 ceph-mon[57919]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:14:14.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:13 smithi083 ceph-mon[64844]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:14:14.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:13 smithi083 ceph-mon[67383]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:14:15.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:14 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-04T16:14:15.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:14 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-04T16:14:15.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:14 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-04T16:14:15.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:14:15 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:14:15] "GET /metrics HTTP/1.1" 200 38535 "" "Prometheus/2.33.4" 2024-03-04T16:14:16.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:15 smithi086 ceph-mon[57919]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:14:16.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:15 smithi086 ceph-mon[57919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-04T16:14:16.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:15 smithi086 ceph-mon[57919]: Upgrade: osd.7 is safe to restart 2024-03-04T16:14:16.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:15 smithi083 ceph-mon[67383]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:14:16.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:15 smithi083 ceph-mon[67383]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-04T16:14:16.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:15 smithi083 ceph-mon[67383]: Upgrade: osd.7 is safe to restart 2024-03-04T16:14:16.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:15 smithi083 ceph-mon[64844]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:14:16.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:15 smithi083 ceph-mon[64844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-04T16:14:16.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:15 smithi083 ceph-mon[64844]: Upgrade: osd.7 is safe to restart 2024-03-04T16:14:17.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:17 smithi086 ceph-mon[57919]: Upgrade: Updating osd.7 2024-03-04T16:14:17.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:17 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:17.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:17 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-04T16:14:17.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:17 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:17.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:17 smithi086 ceph-mon[57919]: Deploying daemon osd.7 on smithi086 2024-03-04T16:14:17.985 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:17 smithi086 ceph-mon[57919]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:14:17.987 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:17 smithi083 ceph-mon[67383]: Upgrade: Updating osd.7 2024-03-04T16:14:17.987 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:17 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:17.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:17 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-04T16:14:17.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:17 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:17.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:17 smithi083 ceph-mon[67383]: Deploying daemon osd.7 on smithi086 2024-03-04T16:14:17.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:17 smithi083 ceph-mon[67383]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:14:17.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:17 smithi083 ceph-mon[64844]: Upgrade: Updating osd.7 2024-03-04T16:14:17.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:17 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:17.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:17 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-04T16:14:17.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:17 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:17.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:17 smithi083 ceph-mon[64844]: Deploying daemon osd.7 on smithi086 2024-03-04T16:14:17.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:17 smithi083 ceph-mon[64844]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:14:20.062 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:19 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:20.062 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:19 smithi086 ceph-mon[57919]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:14:20.063 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:19 smithi086 systemd[1]: Stopping Ceph osd.7 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:14:20.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:19 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:20.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:19 smithi083 ceph-mon[67383]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:14:20.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:19 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:20.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:19 smithi083 ceph-mon[64844]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:14:20.484 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:20 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[37676]: 2024-03-04T16:14:20.060+0000 7fc4817e2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:14:20.484 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:20 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[37676]: 2024-03-04T16:14:20.060+0000 7fc4817e2700 -1 osd.7 141 *** Got signal Terminated *** 2024-03-04T16:14:20.484 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:20 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[37676]: 2024-03-04T16:14:20.060+0000 7fc4817e2700 -1 osd.7 141 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:14:21.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:20 smithi086 ceph-mon[57919]: osd.7 marked itself down and dead 2024-03-04T16:14:21.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:20 smithi083 ceph-mon[67383]: osd.7 marked itself down and dead 2024-03-04T16:14:21.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:20 smithi083 ceph-mon[64844]: osd.7 marked itself down and dead 2024-03-04T16:14:21.612 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:21 smithi086 podman[67754]: 2024-03-04 16:14:21.335791668 +0000 UTC m=+1.380864454 container died 390e1798c738974b78562646ef63b230a35e714830c095a49fb352590c1d609c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., version=8, vcs-type=git, com.redhat.component=centos-stream-container, name=centos-stream, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, ceph=True, release=754, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-04T16:14:21.924 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:21 smithi086 podman[67754]: 2024-03-04 16:14:21.703129351 +0000 UTC m=+1.748202132 container cleanup 390e1798c738974b78562646ef63b230a35e714830c095a49fb352590c1d609c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., io.openshift.expose-services=, version=8, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-04T16:14:21.925 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:21 smithi086 bash[67754]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7 2024-03-04T16:14:22.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:21 smithi086 ceph-mon[57919]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:14:22.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:21 smithi086 ceph-mon[57919]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:14:22.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:21 smithi086 ceph-mon[57919]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-04T16:14:22.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:21 smithi086 ceph-mon[57919]: osdmap e142: 8 total, 7 up, 8 in 2024-03-04T16:14:22.234 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:22 smithi086 podman[67782]: 2024-03-04 16:14:22.013300788 +0000 UTC m=+0.674829684 container remove 390e1798c738974b78562646ef63b230a35e714830c095a49fb352590c1d609c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64) 2024-03-04T16:14:22.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:21 smithi083 ceph-mon[64844]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:14:22.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:21 smithi083 ceph-mon[64844]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:14:22.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:21 smithi083 ceph-mon[64844]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-04T16:14:22.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:21 smithi083 ceph-mon[64844]: osdmap e142: 8 total, 7 up, 8 in 2024-03-04T16:14:22.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:21 smithi083 ceph-mon[67383]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:14:22.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:21 smithi083 ceph-mon[67383]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-04T16:14:22.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:21 smithi083 ceph-mon[67383]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-04T16:14:22.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:21 smithi083 ceph-mon[67383]: osdmap e142: 8 total, 7 up, 8 in 2024-03-04T16:14:22.629 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:22 smithi086 podman[67876]: 2024-03-04 16:14:22.272137754 +0000 UTC m=+0.018337933 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:14:22.629 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:22 smithi086 podman[67876]: 2024-03-04 16:14:22.372592263 +0000 UTC m=+0.118792438 container create 2746b9214e6643df8a8b49730db60e6834701eebd1f5c51ffcf4bc5b29bb3fd5 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-deactivate, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-04T16:14:22.629 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:22 smithi086 podman[67876]: 2024-03-04 16:14:22.629220523 +0000 UTC m=+0.375420704 container init 2746b9214e6643df8a8b49730db60e6834701eebd1f5c51ffcf4bc5b29bb3fd5 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-04T16:14:22.923 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:22 smithi086 podman[67876]: 2024-03-04 16:14:22.632741448 +0000 UTC m=+0.378941621 container start 2746b9214e6643df8a8b49730db60e6834701eebd1f5c51ffcf4bc5b29bb3fd5 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-04T16:14:22.923 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:22 smithi086 podman[67876]: 2024-03-04 16:14:22.72496366 +0000 UTC m=+0.471163838 container attach 2746b9214e6643df8a8b49730db60e6834701eebd1f5c51ffcf4bc5b29bb3fd5 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:14:23.229 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:22 smithi086 ceph-mon[57919]: osdmap e143: 8 total, 7 up, 8 in 2024-03-04T16:14:23.229 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:22 smithi086 podman[67876]: 2024-03-04 16:14:22.923428514 +0000 UTC m=+0.669628692 container died 2746b9214e6643df8a8b49730db60e6834701eebd1f5c51ffcf4bc5b29bb3fd5 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:14:23.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:22 smithi083 ceph-mon[64844]: osdmap e143: 8 total, 7 up, 8 in 2024-03-04T16:14:23.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:22 smithi083 ceph-mon[67383]: osdmap e143: 8 total, 7 up, 8 in 2024-03-04T16:14:23.938 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:23 smithi086 podman[67901]: 2024-03-04 16:14:23.56268874 +0000 UTC m=+0.634361361 container remove 2746b9214e6643df8a8b49730db60e6834701eebd1f5c51ffcf4bc5b29bb3fd5 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:14:23.939 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:23 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-04T16:14:23.939 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:23 smithi086 systemd[1]: Stopped Ceph osd.7 for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:14:23.939 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:23 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.7.service: Consumed 10.612s CPU time. 2024-03-04T16:14:23.939 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:23 smithi086 systemd[1]: Starting Ceph osd.7 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:14:24.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:23 smithi086 ceph-mon[57919]: pgmap v203: 161 pgs: 33 peering, 16 stale+active+clean, 112 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:14:24.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:23 smithi086 ceph-mon[57919]: Health check failed: Reduced data availability: 4 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-04T16:14:24.235 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 podman[68004]: 2024-03-04 16:14:23.909563845 +0000 UTC m=+0.018416824 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:14:24.235 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 podman[68004]: 2024-03-04 16:14:24.038869506 +0000 UTC m=+0.147722473 container create cb4972dcd78729bf46eada043a09864cc0321e053a2fd30aa83d0e81bc282b54 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-activate, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-04T16:14:24.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:23 smithi083 ceph-mon[67383]: pgmap v203: 161 pgs: 33 peering, 16 stale+active+clean, 112 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:14:24.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:23 smithi083 ceph-mon[67383]: Health check failed: Reduced data availability: 4 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-04T16:14:24.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:23 smithi083 ceph-mon[64844]: pgmap v203: 161 pgs: 33 peering, 16 stale+active+clean, 112 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:14:24.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:23 smithi083 ceph-mon[64844]: Health check failed: Reduced data availability: 4 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-04T16:14:24.711 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 podman[68004]: 2024-03-04 16:14:24.267114786 +0000 UTC m=+0.375967769 container init cb4972dcd78729bf46eada043a09864cc0321e053a2fd30aa83d0e81bc282b54 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-activate, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-04T16:14:24.711 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 podman[68004]: 2024-03-04 16:14:24.270655894 +0000 UTC m=+0.379508872 container start cb4972dcd78729bf46eada043a09864cc0321e053a2fd30aa83d0e81bc282b54 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-04T16:14:24.711 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 podman[68004]: 2024-03-04 16:14:24.349457972 +0000 UTC m=+0.458310940 container attach cb4972dcd78729bf46eada043a09864cc0321e053a2fd30aa83d0e81bc282b54 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-activate, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-03-04T16:14:24.984 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-activate[68018]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-04T16:14:24.984 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 bash[68004]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-04T16:14:24.984 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 bash[68004]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-04T16:14:24.984 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 bash[68004]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-04T16:14:24.984 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 bash[68004]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-04T16:14:24.984 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 bash[68004]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-04T16:14:24.985 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 bash[68004]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-04T16:14:24.985 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 bash[68004]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-04T16:14:24.985 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-activate[68018]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-04T16:14:24.985 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-activate[68018]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-04T16:14:24.985 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-activate[68018]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-04T16:14:24.985 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-activate[68018]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-04T16:14:24.985 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-activate[68018]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-04T16:14:24.985 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-activate[68018]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-04T16:14:24.985 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:24 smithi086 podman[68004]: 2024-03-04 16:14:24.735765806 +0000 UTC m=+0.844618780 container died cb4972dcd78729bf46eada043a09864cc0321e053a2fd30aa83d0e81bc282b54 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-activate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:14:25.478 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:24 smithi086 ceph-mon[57919]: Health check failed: Degraded data redundancy: 1/630 objects degraded (0.159%), 1 pg degraded (PG_DEGRADED) 2024-03-04T16:14:25.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:24 smithi083 ceph-mon[64844]: Health check failed: Degraded data redundancy: 1/630 objects degraded (0.159%), 1 pg degraded (PG_DEGRADED) 2024-03-04T16:14:25.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:24 smithi083 ceph-mon[67383]: Health check failed: Degraded data redundancy: 1/630 objects degraded (0.159%), 1 pg degraded (PG_DEGRADED) 2024-03-04T16:14:25.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:25 smithi086 podman[68152]: 2024-03-04 16:14:25.479292957 +0000 UTC m=+0.738277170 container remove cb4972dcd78729bf46eada043a09864cc0321e053a2fd30aa83d0e81bc282b54 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-activate, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:14:25.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:25 smithi086 podman[68218]: 2024-03-04 16:14:25.599655503 +0000 UTC m=+0.019091237 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:14:25.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:25 smithi086 podman[68218]: 2024-03-04 16:14:25.705149892 +0000 UTC m=+0.124585609 container create 881253b7c5f18f8f64125d5abac22cb37849a96140b4ccb176fd97b379a4980d (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5) 2024-03-04T16:14:25.987 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:14:25 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:14:25] "GET /metrics HTTP/1.1" 200 38535 "" "Prometheus/2.33.4" 2024-03-04T16:14:26.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:25 smithi086 ceph-mon[57919]: pgmap v204: 161 pgs: 1 active+undersized+degraded, 33 peering, 11 stale+active+clean, 10 active+undersized, 106 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1/630 objects degraded (0.159%) 2024-03-04T16:14:26.234 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:25 smithi086 podman[68218]: 2024-03-04 16:14:25.907412987 +0000 UTC m=+0.326848713 container init 881253b7c5f18f8f64125d5abac22cb37849a96140b4ccb176fd97b379a4980d (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:14:26.234 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:25 smithi086 podman[68218]: 2024-03-04 16:14:25.910344069 +0000 UTC m=+0.329779796 container start 881253b7c5f18f8f64125d5abac22cb37849a96140b4ccb176fd97b379a4980d (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-04T16:14:26.234 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:25 smithi086 ceph-osd[68237]: -- 172.21.15.86:0/3076236795 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562aa55cf680 con 0x562aa5598c00 2024-03-04T16:14:26.234 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:26 smithi086 bash[68218]: 881253b7c5f18f8f64125d5abac22cb37849a96140b4ccb176fd97b379a4980d 2024-03-04T16:14:26.234 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:26 smithi086 systemd[1]: Started Ceph osd.7 for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:14:26.487 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:25 smithi083 ceph-mon[67383]: pgmap v204: 161 pgs: 1 active+undersized+degraded, 33 peering, 11 stale+active+clean, 10 active+undersized, 106 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1/630 objects degraded (0.159%) 2024-03-04T16:14:26.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:25 smithi083 ceph-mon[64844]: pgmap v204: 161 pgs: 1 active+undersized+degraded, 33 peering, 11 stale+active+clean, 10 active+undersized, 106 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1/630 objects degraded (0.159%) 2024-03-04T16:14:26.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:26 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:14:26.452+0000 7fa5af9817c0 -1 Falling back to public interface 2024-03-04T16:14:27.194 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:27.194 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:27.194 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:27 smithi086 ceph-mon[57919]: pgmap v205: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-03-04T16:14:27.194 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:27.194 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:27 smithi083 ceph-mon[67383]: pgmap v205: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-03-04T16:14:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:27.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:27.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:27 smithi083 ceph-mon[64844]: pgmap v205: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-03-04T16:14:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:28.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:28 smithi086 ceph-mon[57919]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 7 pgs peering) 2024-03-04T16:14:28.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:28 smithi083 ceph-mon[64844]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 7 pgs peering) 2024-03-04T16:14:28.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:28 smithi083 ceph-mon[67383]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 7 pgs peering) 2024-03-04T16:14:28.613 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:14:29.259 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:14:29.260 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (7m) 2m ago 12m 24.9M - 0.23.0 ba2b418f427c 3c6dc3779a0e 2024-03-04T16:14:29.260 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (6m) 33s ago 12m 54.1M - 8.3.5 dad864ee21e9 fb754d58c53f 2024-03-04T16:14:29.260 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 running (6m) 2m ago 11m 53.6M - 3.5 e1d6a67b021e d58e71ccf9db 2024-03-04T16:14:29.260 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443,9283,8765 running (5m) 33s ago 16m 452M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c33100cb8a8 2024-03-04T16:14:29.260 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:8443,9283,8765 running (6m) 2m ago 18m 506M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 164dd93609ee 2024-03-04T16:14:29.260 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (5m) 2m ago 18m 48.8M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c173a2442197 2024-03-04T16:14:29.260 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (5m) 33s ago 17m 40.0M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 71557985ed5a 2024-03-04T16:14:29.260 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (5m) 2m ago 17m 40.6M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e eff1982ba275 2024-03-04T16:14:29.260 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (7m) 2m ago 13m 17.9M - 1.3.1 1dbe0e931976 9bf54b460640 2024-03-04T16:14:29.260 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (6m) 33s ago 12m 15.6M - 1.3.1 1dbe0e931976 b72dd80b2588 2024-03-04T16:14:29.260 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (4m) 2m ago 16m 44.4M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e f7ecdc5b1520 2024-03-04T16:14:29.260 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (3m) 2m ago 16m 46.6M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 2c648d9a2b38 2024-03-04T16:14:29.260 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (2m) 2m ago 15m 53.0M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c971c1f3f0a 2024-03-04T16:14:29.261 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (2m) 2m ago 15m 11.6M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 8991d467d41c 2024-03-04T16:14:29.261 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (108s) 33s ago 14m 55.2M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 912383028c72 2024-03-04T16:14:29.261 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (73s) 33s ago 14m 50.3M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 53215200f312 2024-03-04T16:14:29.261 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (38s) 33s ago 14m 13.5M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e a4584e5b3044 2024-03-04T16:14:29.261 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 starting - - - 2992M 2024-03-04T16:14:29.261 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 running (6m) 33s ago 13m 71.5M - 2.33.4 514e6a882f6e b563a246eeac 2024-03-04T16:14:29.261 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (11m) 2m ago 11m 74.5M - 17.2.0 e1d6a67b021e 65f14042f354 2024-03-04T16:14:29.261 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (11m) 33s ago 11m 76.0M - 17.2.0 e1d6a67b021e 78175cc6dc5d 2024-03-04T16:14:29.261 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (11m) 2m ago 11m 72.6M - 17.2.0 e1d6a67b021e 01dcf858bd62 2024-03-04T16:14:29.261 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (11m) 33s ago 11m 78.1M - 17.2.0 e1d6a67b021e 8f70a5ce9f0d 2024-03-04T16:14:29.345 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:14:29.169+0000 7fa5af9817c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-03-04T16:14:29.346 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:14:29.210+0000 7fa5af9817c0 -1 osd.7 141 log_to_monitors true 2024-03-04T16:14:29.346 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:29 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:29.346 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:29 smithi086 ceph-mon[57919]: from='client.44538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:14:29.346 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:29 smithi086 ceph-mon[57919]: pgmap v206: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-03-04T16:14:29.346 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:29 smithi086 ceph-mon[57919]: from='client.54475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:14:29.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:29 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:29.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:29 smithi083 ceph-mon[64844]: from='client.44538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:14:29.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:29 smithi083 ceph-mon[64844]: pgmap v206: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-03-04T16:14:29.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:29 smithi083 ceph-mon[64844]: from='client.54475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:14:29.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:29 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:29.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:29 smithi083 ceph-mon[67383]: from='client.44538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:14:29.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:29 smithi083 ceph-mon[67383]: pgmap v206: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-03-04T16:14:29.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:29 smithi083 ceph-mon[67383]: from='client.54475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:14:29.649 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:14:29.649 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:14:29.649 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 3 2024-03-04T16:14:29.649 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:14:29.649 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:14:29.649 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:14:29.649 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:14:29.650 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:14:29.650 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 7 2024-03-04T16:14:29.650 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:14:29.650 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:14:29.650 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-04T16:14:29.650 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:14:29.650 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:14:29.650 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-04T16:14:29.650 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 12 2024-03-04T16:14:29.650 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:14:29.650 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:14:29.979 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:14:29.979 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:14:29.979 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:14:29.979 INFO:teuthology.orchestra.run.smithi083.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:14:29.979 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [ 2024-03-04T16:14:29.979 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr", 2024-03-04T16:14:29.979 INFO:teuthology.orchestra.run.smithi083.stdout: "mon" 2024-03-04T16:14:29.979 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-04T16:14:29.980 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "12/23 daemons upgraded", 2024-03-04T16:14:29.980 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "Currently upgrading osd daemons", 2024-03-04T16:14:29.980 INFO:teuthology.orchestra.run.smithi083.stdout: "is_paused": false 2024-03-04T16:14:29.980 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:14:30.182 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:30 smithi086 ceph-mon[57919]: from='osd.7 [v2:172.21.15.86:6824/692633992,v1:172.21.15.86:6825/692633992]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T16:14:30.183 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:30 smithi086 ceph-mon[57919]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T16:14:30.183 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:30 smithi086 ceph-mon[57919]: from='client.44550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:14:30.183 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:30 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/4076813910' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:30.183 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:30 smithi086 ceph-mon[57919]: from='client.54496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:14:30.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:30 smithi083 ceph-mon[67383]: from='osd.7 [v2:172.21.15.86:6824/692633992,v1:172.21.15.86:6825/692633992]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T16:14:30.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:30 smithi083 ceph-mon[67383]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T16:14:30.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:30 smithi083 ceph-mon[67383]: from='client.44550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:14:30.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:30 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/4076813910' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:30.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:30 smithi083 ceph-mon[67383]: from='client.54496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:14:30.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:30 smithi083 ceph-mon[64844]: from='osd.7 [v2:172.21.15.86:6824/692633992,v1:172.21.15.86:6825/692633992]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T16:14:30.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:30 smithi083 ceph-mon[64844]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T16:14:30.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:30 smithi083 ceph-mon[64844]: from='client.44550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:14:30.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:30 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/4076813910' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:30.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:30 smithi083 ceph-mon[64844]: from='client.54496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:14:30.375 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 83/630 objects degraded (13.175%), 25 pgs degraded 2024-03-04T16:14:30.375 INFO:teuthology.orchestra.run.smithi083.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-04T16:14:30.375 INFO:teuthology.orchestra.run.smithi083.stdout: osd.7 (root=default,host=smithi086) is down 2024-03-04T16:14:30.375 INFO:teuthology.orchestra.run.smithi083.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-03-04T16:14:30.375 INFO:teuthology.orchestra.run.smithi083.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-03-04T16:14:30.375 INFO:teuthology.orchestra.run.smithi083.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 83/630 objects degraded (13.175%), 25 pgs degraded 2024-03-04T16:14:30.376 INFO:teuthology.orchestra.run.smithi083.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-03-04T16:14:30.376 INFO:teuthology.orchestra.run.smithi083.stdout: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-03-04T16:14:30.376 INFO:teuthology.orchestra.run.smithi083.stdout: pg 2.7 is active+undersized+degraded, acting [6,2] 2024-03-04T16:14:30.376 INFO:teuthology.orchestra.run.smithi083.stdout: pg 2.8 is active+undersized+degraded, acting [0,1] 2024-03-04T16:14:30.376 INFO:teuthology.orchestra.run.smithi083.stdout: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-03-04T16:14:30.376 INFO:teuthology.orchestra.run.smithi083.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-03-04T16:14:30.376 INFO:teuthology.orchestra.run.smithi083.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-03-04T16:14:30.377 INFO:teuthology.orchestra.run.smithi083.stdout: pg 2.12 is active+undersized+degraded, acting [5,3] 2024-03-04T16:14:30.377 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-03-04T16:14:30.377 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.8 is active+undersized+degraded, acting [3,1] 2024-03-04T16:14:30.377 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.9 is active+undersized+degraded, acting [4,2] 2024-03-04T16:14:30.377 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.d is active+undersized+degraded, acting [5,6] 2024-03-04T16:14:30.377 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.e is active+undersized+degraded, acting [4,1] 2024-03-04T16:14:30.377 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2024-03-04T16:14:30.377 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.11 is active+undersized+degraded, acting [4,6] 2024-03-04T16:14:30.377 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.12 is active+undersized+degraded, acting [0,3] 2024-03-04T16:14:30.378 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.13 is active+undersized+degraded, acting [4,2] 2024-03-04T16:14:30.378 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.14 is active+undersized+degraded, acting [4,6] 2024-03-04T16:14:30.378 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-03-04T16:14:30.378 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.16 is active+undersized+degraded, acting [5,1] 2024-03-04T16:14:30.378 INFO:teuthology.orchestra.run.smithi083.stdout: pg 3.1b is active+undersized+degraded, acting [0,4] 2024-03-04T16:14:30.378 INFO:teuthology.orchestra.run.smithi083.stdout: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-03-04T16:14:30.378 INFO:teuthology.orchestra.run.smithi083.stdout: pg 4.3 is active+undersized+degraded, acting [0,5] 2024-03-04T16:14:30.378 INFO:teuthology.orchestra.run.smithi083.stdout: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-03-04T16:14:30.378 INFO:teuthology.orchestra.run.smithi083.stdout: pg 4.15 is active+undersized+degraded, acting [5,3] 2024-03-04T16:14:30.984 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:14:30 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:14:30.571+0000 7fa59cdd1700 -1 osd.7 141 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:14:31.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:31 smithi086 ceph-mon[57919]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-04T16:14:31.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:31 smithi086 ceph-mon[57919]: from='osd.7 [v2:172.21.15.86:6824/692633992,v1:172.21.15.86:6825/692633992]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:14:31.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:31 smithi086 ceph-mon[57919]: osdmap e144: 8 total, 7 up, 8 in 2024-03-04T16:14:31.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:31 smithi086 ceph-mon[57919]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:14:31.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:31 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/2128096652' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:14:31.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:31 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:31.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:31 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:31.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:31 smithi086 ceph-mon[57919]: from='osd.7 ' entity='osd.7' 2024-03-04T16:14:31.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:31 smithi086 ceph-mon[57919]: pgmap v208: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-03-04T16:14:31.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:31 smithi083 ceph-mon[64844]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-04T16:14:31.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:31 smithi083 ceph-mon[64844]: from='osd.7 [v2:172.21.15.86:6824/692633992,v1:172.21.15.86:6825/692633992]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:14:31.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:31 smithi083 ceph-mon[64844]: osdmap e144: 8 total, 7 up, 8 in 2024-03-04T16:14:31.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:31 smithi083 ceph-mon[64844]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:14:31.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:31 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/2128096652' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:14:31.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:31 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:31.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:31 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:31.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:31 smithi083 ceph-mon[64844]: from='osd.7 ' entity='osd.7' 2024-03-04T16:14:31.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:31 smithi083 ceph-mon[64844]: pgmap v208: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-03-04T16:14:31.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:31 smithi083 ceph-mon[67383]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-04T16:14:31.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:31 smithi083 ceph-mon[67383]: from='osd.7 [v2:172.21.15.86:6824/692633992,v1:172.21.15.86:6825/692633992]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:14:31.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:31 smithi083 ceph-mon[67383]: osdmap e144: 8 total, 7 up, 8 in 2024-03-04T16:14:31.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:31 smithi083 ceph-mon[67383]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T16:14:31.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:31 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/2128096652' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:14:31.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:31 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:31.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:31 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:31.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:31 smithi083 ceph-mon[67383]: from='osd.7 ' entity='osd.7' 2024-03-04T16:14:31.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:31 smithi083 ceph-mon[67383]: pgmap v208: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-03-04T16:14:32.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:32 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:32.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:32 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:32.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:32 smithi086 ceph-mon[57919]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:14:32.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:32 smithi086 ceph-mon[57919]: osd.7 [v2:172.21.15.86:6824/692633992,v1:172.21.15.86:6825/692633992] boot 2024-03-04T16:14:32.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:32 smithi086 ceph-mon[57919]: osdmap e145: 8 total, 8 up, 8 in 2024-03-04T16:14:32.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:32 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:14:32.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:32 smithi086 ceph-mon[57919]: Health check update: Degraded data redundancy: 83/630 objects degraded (13.175%), 25 pgs degraded (PG_DEGRADED) 2024-03-04T16:14:32.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:32 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:32.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:32 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:32.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:32 smithi083 ceph-mon[64844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:14:32.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:32 smithi083 ceph-mon[64844]: osd.7 [v2:172.21.15.86:6824/692633992,v1:172.21.15.86:6825/692633992] boot 2024-03-04T16:14:32.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:32 smithi083 ceph-mon[64844]: osdmap e145: 8 total, 8 up, 8 in 2024-03-04T16:14:32.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:32 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:14:32.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:32 smithi083 ceph-mon[64844]: Health check update: Degraded data redundancy: 83/630 objects degraded (13.175%), 25 pgs degraded (PG_DEGRADED) 2024-03-04T16:14:32.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:32 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:32.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:32 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:32.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:32 smithi083 ceph-mon[67383]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-04T16:14:32.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:32 smithi083 ceph-mon[67383]: osd.7 [v2:172.21.15.86:6824/692633992,v1:172.21.15.86:6825/692633992] boot 2024-03-04T16:14:32.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:32 smithi083 ceph-mon[67383]: osdmap e145: 8 total, 8 up, 8 in 2024-03-04T16:14:32.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:32 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:14:32.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:32 smithi083 ceph-mon[67383]: Health check update: Degraded data redundancy: 83/630 objects degraded (13.175%), 25 pgs degraded (PG_DEGRADED) 2024-03-04T16:14:33.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:33 smithi086 ceph-mon[57919]: osdmap e146: 8 total, 8 up, 8 in 2024-03-04T16:14:33.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:33 smithi086 ceph-mon[57919]: pgmap v211: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 83/630 objects degraded (13.175%) 2024-03-04T16:14:33.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:33 smithi083 ceph-mon[64844]: osdmap e146: 8 total, 8 up, 8 in 2024-03-04T16:14:33.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:33 smithi083 ceph-mon[64844]: pgmap v211: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 83/630 objects degraded (13.175%) 2024-03-04T16:14:33.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:33 smithi083 ceph-mon[67383]: osdmap e146: 8 total, 8 up, 8 in 2024-03-04T16:14:33.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:33 smithi083 ceph-mon[67383]: pgmap v211: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 83/630 objects degraded (13.175%) 2024-03-04T16:14:35.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:14:35] "GET /metrics HTTP/1.1" 200 38617 "" "Prometheus/2.33.4" 2024-03-04T16:14:35.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:35.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: pgmap v212: 161 pgs: 24 active+undersized+degraded, 39 active+undersized, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 82/630 objects degraded (13.016%) 2024-03-04T16:14:35.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:35.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:35.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:14:35.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:35.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:14:35.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:35.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:35.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:35.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:35.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:35.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-04T16:14:35.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-04T16:14:35.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-04T16:14:35.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-04T16:14:35.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-04T16:14:35.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-04T16:14:35.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-04T16:14:35.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-04T16:14:35.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-04T16:14:35.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-04T16:14:35.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-04T16:14:35.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-04T16:14:35.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:35.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: pgmap v212: 161 pgs: 24 active+undersized+degraded, 39 active+undersized, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 82/630 objects degraded (13.016%) 2024-03-04T16:14:35.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:35.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:35.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:14:35.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:35.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:14:35.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:35.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:35.991 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:35.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:35.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:35.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-04T16:14:35.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-04T16:14:35.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-04T16:14:35.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-04T16:14:35.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-04T16:14:35.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-04T16:14:35.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-04T16:14:35.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-04T16:14:35.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-04T16:14:35.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-04T16:14:35.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-04T16:14:35.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-04T16:14:35.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-04T16:14:35.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-04T16:14:35.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-04T16:14:35.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-04T16:14:35.993 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-04T16:14:35.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-04T16:14:35.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-04T16:14:35.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-04T16:14:35.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-04T16:14:35.993 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:35 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-04T16:14:36.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:36.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: pgmap v212: 161 pgs: 24 active+undersized+degraded, 39 active+undersized, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 82/630 objects degraded (13.016%) 2024-03-04T16:14:36.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:36.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:36.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:14:36.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:36.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:14:36.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:36.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:36.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:36.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:36.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:36.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-04T16:14:36.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-04T16:14:36.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-04T16:14:36.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-04T16:14:36.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-04T16:14:36.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-04T16:14:36.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-04T16:14:36.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-04T16:14:36.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-04T16:14:36.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-04T16:14:36.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-04T16:14:36.236 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-04T16:14:36.237 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-04T16:14:36.237 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-04T16:14:36.237 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-04T16:14:36.237 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-04T16:14:36.237 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:35 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-04T16:14:36.966 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:36 smithi083 ceph-mon[64844]: Upgrade: Setting container_image for all osd 2024-03-04T16:14:36.966 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:36 smithi083 ceph-mon[64844]: Upgrade: Setting require_osd_release to 19 squid 2024-03-04T16:14:36.967 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:36 smithi083 ceph-mon[64844]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-04T16:14:36.967 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:36 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-04T16:14:36.967 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:36 smithi083 ceph-mon[64844]: osdmap e147: 8 total, 8 up, 8 in 2024-03-04T16:14:36.967 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:36 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:36.967 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:36 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:36.967 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:36 smithi083 ceph-mon[67383]: Upgrade: Setting container_image for all osd 2024-03-04T16:14:36.967 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:36 smithi083 ceph-mon[67383]: Upgrade: Setting require_osd_release to 19 squid 2024-03-04T16:14:36.967 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:36 smithi083 ceph-mon[67383]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-04T16:14:36.968 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:36 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-04T16:14:36.968 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:36 smithi083 ceph-mon[67383]: osdmap e147: 8 total, 8 up, 8 in 2024-03-04T16:14:36.968 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:36 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:36.968 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:36 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:37.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:36 smithi086 ceph-mon[57919]: Upgrade: Setting container_image for all osd 2024-03-04T16:14:37.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:36 smithi086 ceph-mon[57919]: Upgrade: Setting require_osd_release to 19 squid 2024-03-04T16:14:37.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:36 smithi086 ceph-mon[57919]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-04T16:14:37.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:36 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-04T16:14:37.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:36 smithi086 ceph-mon[57919]: osdmap e147: 8 total, 8 up, 8 in 2024-03-04T16:14:37.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:36 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:14:37.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:36 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:37.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:37 smithi083 ceph-mon[64844]: Upgrade: Setting container_image for all mds 2024-03-04T16:14:37.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:37 smithi083 ceph-mon[64844]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-04T16:14:37.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:37 smithi083 ceph-mon[64844]: Health check update: Degraded data redundancy: 82/630 objects degraded (13.016%), 24 pgs degraded (PG_DEGRADED) 2024-03-04T16:14:37.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:37 smithi083 ceph-mon[64844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 82/630 objects degraded (13.016%), 24 pgs degraded) 2024-03-04T16:14:37.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:37 smithi083 ceph-mon[64844]: Cluster is now healthy 2024-03-04T16:14:37.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:37 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:37.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:37 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi083.szjzkw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:14:37.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:37 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:37.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:37 smithi083 ceph-mon[67383]: Upgrade: Setting container_image for all mds 2024-03-04T16:14:37.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:37 smithi083 ceph-mon[67383]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-04T16:14:37.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:37 smithi083 ceph-mon[67383]: Health check update: Degraded data redundancy: 82/630 objects degraded (13.016%), 24 pgs degraded (PG_DEGRADED) 2024-03-04T16:14:37.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:37 smithi083 ceph-mon[67383]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 82/630 objects degraded (13.016%), 24 pgs degraded) 2024-03-04T16:14:37.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:37 smithi083 ceph-mon[67383]: Cluster is now healthy 2024-03-04T16:14:37.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:37 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:37.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:37 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi083.szjzkw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:14:37.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:37 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:38.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:37 smithi086 ceph-mon[57919]: Upgrade: Setting container_image for all mds 2024-03-04T16:14:38.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:37 smithi086 ceph-mon[57919]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-04T16:14:38.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:37 smithi086 ceph-mon[57919]: Health check update: Degraded data redundancy: 82/630 objects degraded (13.016%), 24 pgs degraded (PG_DEGRADED) 2024-03-04T16:14:38.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:37 smithi086 ceph-mon[57919]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 82/630 objects degraded (13.016%), 24 pgs degraded) 2024-03-04T16:14:38.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:37 smithi086 ceph-mon[57919]: Cluster is now healthy 2024-03-04T16:14:38.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:37 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:38.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:37 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi083.szjzkw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:14:38.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:37 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:38.987 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:38 smithi083 ceph-mon[67383]: Upgrade: Updating rgw.foo.smithi083.szjzkw (1/4) 2024-03-04T16:14:38.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:38 smithi083 ceph-mon[67383]: Deploying daemon rgw.foo.smithi083.szjzkw on smithi083 2024-03-04T16:14:38.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:38 smithi083 ceph-mon[64844]: Upgrade: Updating rgw.foo.smithi083.szjzkw (1/4) 2024-03-04T16:14:38.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:38 smithi083 ceph-mon[64844]: Deploying daemon rgw.foo.smithi083.szjzkw on smithi083 2024-03-04T16:14:39.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:38 smithi086 ceph-mon[57919]: Upgrade: Updating rgw.foo.smithi083.szjzkw (1/4) 2024-03-04T16:14:39.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:38 smithi086 ceph-mon[57919]: Deploying daemon rgw.foo.smithi083.szjzkw on smithi083 2024-03-04T16:14:39.925 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:39 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:39.925 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:39 smithi083 ceph-mon[64844]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 701 B/s rd, 0 op/s 2024-03-04T16:14:39.925 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:39 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:39.925 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:39 smithi083 ceph-mon[67383]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 701 B/s rd, 0 op/s 2024-03-04T16:14:40.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:39 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:40.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:39 smithi086 ceph-mon[57919]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 701 B/s rd, 0 op/s 2024-03-04T16:14:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:41 smithi086 ceph-mon[57919]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 616 B/s rd, 0 op/s 2024-03-04T16:14:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:42.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:41 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:42.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:41 smithi083 ceph-mon[64844]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 616 B/s rd, 0 op/s 2024-03-04T16:14:42.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:42.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:42.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:42.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:41 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:42.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:41 smithi083 ceph-mon[67383]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 616 B/s rd, 0 op/s 2024-03-04T16:14:42.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:42.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:42.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:42.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:41 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:43.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:42 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:43.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:42 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:43.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:42 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:44.339 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:43 smithi083 ceph-mon[67383]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:14:44.339 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:43 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:44.339 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:43 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi083.hqmkpf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:14:44.339 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:43 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:44.340 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:43 smithi083 ceph-mon[64844]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:14:44.340 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:43 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:44.340 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:43 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi083.hqmkpf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:14:44.340 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:43 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:44.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:43 smithi086 ceph-mon[57919]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:14:44.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:43 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:44.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:43 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi083.hqmkpf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:14:44.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:43 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:45.266 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:45 smithi083 ceph-mon[64844]: Upgrade: Updating rgw.smpl.smithi083.hqmkpf (2/4) 2024-03-04T16:14:45.267 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:45 smithi083 ceph-mon[64844]: Deploying daemon rgw.smpl.smithi083.hqmkpf on smithi083 2024-03-04T16:14:45.267 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:45 smithi083 ceph-mon[67383]: Upgrade: Updating rgw.smpl.smithi083.hqmkpf (2/4) 2024-03-04T16:14:45.267 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:45 smithi083 ceph-mon[67383]: Deploying daemon rgw.smpl.smithi083.hqmkpf on smithi083 2024-03-04T16:14:45.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:45 smithi086 ceph-mon[57919]: Upgrade: Updating rgw.smpl.smithi083.hqmkpf (2/4) 2024-03-04T16:14:45.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:45 smithi086 ceph-mon[57919]: Deploying daemon rgw.smpl.smithi083.hqmkpf on smithi083 2024-03-04T16:14:45.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:14:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:14:45] "GET /metrics HTTP/1.1" 200 38619 "" "Prometheus/2.33.4" 2024-03-04T16:14:46.265 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:46 smithi083 ceph-mon[64844]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-04T16:14:46.265 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:46 smithi083 ceph-mon[67383]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-04T16:14:46.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:46 smithi086 ceph-mon[57919]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-04T16:14:47.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:47 smithi083 ceph-mon[64844]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 94 B/s wr, 93 op/s 2024-03-04T16:14:47.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:47 smithi083 ceph-mon[67383]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 94 B/s wr, 93 op/s 2024-03-04T16:14:47.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:47 smithi086 ceph-mon[57919]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 94 B/s wr, 93 op/s 2024-03-04T16:14:48.866 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:48 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:48.866 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:48 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:48.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:48 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:48.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:48 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:48.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:48 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:48.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:48 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:49.919 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:49 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:49.919 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:49 smithi086 ceph-mon[57919]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-04T16:14:49.919 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:49 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:49.919 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:49 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.ezumfo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:14:49.919 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:49 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:49.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:49 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:49.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:49 smithi083 ceph-mon[64844]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-04T16:14:49.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:49 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:49.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:49 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.ezumfo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:14:49.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:49 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:49.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:49 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:49.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:49 smithi083 ceph-mon[67383]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-04T16:14:49.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:49 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:49.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:49 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.ezumfo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:14:49.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:49 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:50.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:50 smithi086 ceph-mon[57919]: Upgrade: Updating rgw.foo.smithi086.ezumfo (3/4) 2024-03-04T16:14:50.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:50 smithi086 ceph-mon[57919]: Deploying daemon rgw.foo.smithi086.ezumfo on smithi086 2024-03-04T16:14:50.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:50 smithi083 ceph-mon[67383]: Upgrade: Updating rgw.foo.smithi086.ezumfo (3/4) 2024-03-04T16:14:50.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:50 smithi083 ceph-mon[67383]: Deploying daemon rgw.foo.smithi086.ezumfo on smithi086 2024-03-04T16:14:50.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:50 smithi083 ceph-mon[64844]: Upgrade: Updating rgw.foo.smithi086.ezumfo (3/4) 2024-03-04T16:14:50.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:50 smithi083 ceph-mon[64844]: Deploying daemon rgw.foo.smithi086.ezumfo on smithi086 2024-03-04T16:14:51.912 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:51 smithi086 ceph-mon[57919]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 85 B/s wr, 129 op/s 2024-03-04T16:14:51.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:51 smithi083 ceph-mon[64844]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 85 B/s wr, 129 op/s 2024-03-04T16:14:51.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:51 smithi083 ceph-mon[67383]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 85 B/s wr, 129 op/s 2024-03-04T16:14:54.157 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:53 smithi086 ceph-mon[57919]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-04T16:14:54.158 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:53 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:54.158 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:53 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:54.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:53 smithi083 ceph-mon[64844]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-04T16:14:54.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:53 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:54.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:53 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:54.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:53 smithi083 ceph-mon[67383]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-04T16:14:54.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:53 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:54.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:53 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:55.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:55 smithi086 ceph-mon[57919]: Upgrade: Updating rgw.smpl.smithi086.yqigbl (4/4) 2024-03-04T16:14:55.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:55 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:55.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:55 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.yqigbl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:14:55.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:55 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:55.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:55 smithi086 ceph-mon[57919]: Deploying daemon rgw.smpl.smithi086.yqigbl on smithi086 2024-03-04T16:14:55.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:55 smithi086 ceph-mon[57919]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 170 B/s wr, 173 op/s 2024-03-04T16:14:55.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:14:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:14:55] "GET /metrics HTTP/1.1" 200 38619 "" "Prometheus/2.33.4" 2024-03-04T16:14:55.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:55 smithi083 ceph-mon[64844]: Upgrade: Updating rgw.smpl.smithi086.yqigbl (4/4) 2024-03-04T16:14:55.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:55 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:55.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:55 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.yqigbl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:14:55.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:55 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:55.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:55 smithi083 ceph-mon[64844]: Deploying daemon rgw.smpl.smithi086.yqigbl on smithi086 2024-03-04T16:14:55.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:55 smithi083 ceph-mon[64844]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 170 B/s wr, 173 op/s 2024-03-04T16:14:55.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:55 smithi083 ceph-mon[67383]: Upgrade: Updating rgw.smpl.smithi086.yqigbl (4/4) 2024-03-04T16:14:55.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:55 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:14:55.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:55 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.yqigbl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:14:55.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:55 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:55.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:55 smithi083 ceph-mon[67383]: Deploying daemon rgw.smpl.smithi086.yqigbl on smithi086 2024-03-04T16:14:55.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:55 smithi083 ceph-mon[67383]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 170 B/s wr, 173 op/s 2024-03-04T16:14:57.035 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:56 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:57.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:57.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:56 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:57.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:57 smithi086 ceph-mon[57919]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 160 KiB/s rd, 255 B/s wr, 247 op/s 2024-03-04T16:14:58.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:57 smithi083 ceph-mon[64844]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 160 KiB/s rd, 255 B/s wr, 247 op/s 2024-03-04T16:14:58.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:57 smithi083 ceph-mon[67383]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 160 KiB/s rd, 255 B/s wr, 247 op/s 2024-03-04T16:15:00.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:59 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:00.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:59 smithi086 ceph-mon[57919]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-04T16:15:00.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:59 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:00.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:14:59 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:00.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:59 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:00.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:59 smithi083 ceph-mon[67383]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-04T16:15:00.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:59 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:00.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:14:59 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:00.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:59 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:00.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:59 smithi083 ceph-mon[64844]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-04T16:15:00.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:59 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:00.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:14:59 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:00.749 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:15:01.421 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:15:01.421 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (7m) 2m ago 12m 24.9M - 0.23.0 ba2b418f427c 3c6dc3779a0e 2024-03-04T16:15:01.421 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (7m) 30s ago 12m 54.2M - 8.3.5 dad864ee21e9 fb754d58c53f 2024-03-04T16:15:01.421 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 running (6m) 2m ago 11m 53.6M - 3.5 e1d6a67b021e d58e71ccf9db 2024-03-04T16:15:01.421 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443,9283,8765 running (6m) 30s ago 17m 452M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c33100cb8a8 2024-03-04T16:15:01.421 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:8443,9283,8765 running (7m) 2m ago 18m 506M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 164dd93609ee 2024-03-04T16:15:01.421 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (6m) 2m ago 19m 48.8M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c173a2442197 2024-03-04T16:15:01.421 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (5m) 30s ago 17m 41.0M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 71557985ed5a 2024-03-04T16:15:01.421 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (5m) 2m ago 17m 40.6M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e eff1982ba275 2024-03-04T16:15:01.421 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (7m) 2m ago 13m 17.9M - 1.3.1 1dbe0e931976 9bf54b460640 2024-03-04T16:15:01.421 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (7m) 30s ago 13m 15.6M - 1.3.1 1dbe0e931976 b72dd80b2588 2024-03-04T16:15:01.422 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (4m) 2m ago 17m 44.4M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e f7ecdc5b1520 2024-03-04T16:15:01.422 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (4m) 2m ago 16m 46.6M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 2c648d9a2b38 2024-03-04T16:15:01.422 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (3m) 2m ago 16m 53.0M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c971c1f3f0a 2024-03-04T16:15:01.422 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (2m) 2m ago 15m 11.6M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 8991d467d41c 2024-03-04T16:15:01.422 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (2m) 30s ago 15m 56.1M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 912383028c72 2024-03-04T16:15:01.422 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (105s) 30s ago 15m 51.7M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 53215200f312 2024-03-04T16:15:01.422 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (70s) 30s ago 14m 42.9M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e a4584e5b3044 2024-03-04T16:15:01.422 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (35s) 30s ago 14m 14.8M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 881253b7c5f1 2024-03-04T16:15:01.422 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 running (6m) 30s ago 13m 71.6M - 2.33.4 514e6a882f6e b563a246eeac 2024-03-04T16:15:01.422 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 starting - - - - 2024-03-04T16:15:01.422 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 starting - - - - 2024-03-04T16:15:01.422 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 starting - - - - 2024-03-04T16:15:01.422 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 starting - - - - 2024-03-04T16:15:01.820 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:15:01.820 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:15:01.820 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 3 2024-03-04T16:15:01.820 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:15:01.821 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:15:01.821 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:15:01.821 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:15:01.821 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:15:01.821 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 8 2024-03-04T16:15:01.821 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:15:01.821 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:15:01.821 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 4 2024-03-04T16:15:01.821 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:15:01.821 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:15:01.821 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 17 2024-03-04T16:15:01.821 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:15:01.821 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:15:01.987 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:01 smithi083 ceph-mon[67383]: from='client.54592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:01.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:01 smithi083 ceph-mon[67383]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 138 KiB/s rd, 170 B/s wr, 212 op/s 2024-03-04T16:15:01.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:01 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/3471667689' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:01.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:01 smithi083 ceph-mon[64844]: from='client.54592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:01.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:01 smithi083 ceph-mon[64844]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 138 KiB/s rd, 170 B/s wr, 212 op/s 2024-03-04T16:15:01.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:01 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/3471667689' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:02.162 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:15:02.162 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:15:02.162 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:15:02.162 INFO:teuthology.orchestra.run.smithi083.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:15:02.162 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [ 2024-03-04T16:15:02.162 INFO:teuthology.orchestra.run.smithi083.stdout: "osd", 2024-03-04T16:15:02.162 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr", 2024-03-04T16:15:02.162 INFO:teuthology.orchestra.run.smithi083.stdout: "mon" 2024-03-04T16:15:02.162 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-04T16:15:02.162 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "13/23 daemons upgraded", 2024-03-04T16:15:02.162 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "Currently upgrading rgw daemons", 2024-03-04T16:15:02.162 INFO:teuthology.orchestra.run.smithi083.stdout: "is_paused": false 2024-03-04T16:15:02.163 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:15:02.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:01 smithi086 ceph-mon[57919]: from='client.54592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:02.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:01 smithi086 ceph-mon[57919]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 138 KiB/s rd, 170 B/s wr, 212 op/s 2024-03-04T16:15:02.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:01 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/3471667689' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:02.571 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_OK 2024-03-04T16:15:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:02 smithi086 ceph-mon[57919]: from='client.44670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:02 smithi086 ceph-mon[57919]: from='client.54604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:02 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/2420730925' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:15:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:02 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:02 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:02 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:03.235 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:02 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:03.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:02 smithi083 ceph-mon[64844]: from='client.44670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:03.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:02 smithi083 ceph-mon[64844]: from='client.54604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:02 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/2420730925' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:15:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:02 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:02 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:02 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:02 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:03.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:02 smithi083 ceph-mon[67383]: from='client.44670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:03.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:02 smithi083 ceph-mon[67383]: from='client.54604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:03.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:02 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/2420730925' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:15:03.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:02 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:03.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:02 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:03.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:02 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:03.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:02 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:04.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:03 smithi086 ceph-mon[57919]: from='client.54616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:04.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:03 smithi086 ceph-mon[57919]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 255 B/s wr, 206 op/s 2024-03-04T16:15:04.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:03 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:04.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:03 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:04.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:03 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:04.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:03 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:04.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:03 smithi083 ceph-mon[64844]: from='client.54616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:04.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:03 smithi083 ceph-mon[64844]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 255 B/s wr, 206 op/s 2024-03-04T16:15:04.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:03 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:04.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:03 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:04.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:03 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:04.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:03 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:04.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:03 smithi083 ceph-mon[67383]: from='client.54616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:04.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:03 smithi083 ceph-mon[67383]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 255 B/s wr, 206 op/s 2024-03-04T16:15:04.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:03 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:04.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:03 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:04.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:03 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:04.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:03 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:05.838 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:15:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:15:05] "GET /metrics HTTP/1.1" 200 38620 "" "Prometheus/2.33.4" 2024-03-04T16:15:06.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:05 smithi086 ceph-mon[57919]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-04T16:15:06.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:05 smithi083 ceph-mon[64844]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-04T16:15:06.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:05 smithi083 ceph-mon[67383]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-04T16:15:08.233 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 170 B/s wr, 163 op/s 2024-03-04T16:15:08.233 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.233 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.233 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.233 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.234 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:15:08.234 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:15:08.234 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.234 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:15:08.234 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.234 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.234 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.234 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.234 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.234 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.234 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.234 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi083.szjzkw"}]: dispatch 2024-03-04T16:15:08.234 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi083.szjzkw"}]': finished 2024-03-04T16:15:08.235 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi086.ezumfo"}]: dispatch 2024-03-04T16:15:08.235 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi086.ezumfo"}]': finished 2024-03-04T16:15:08.235 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi083.hqmkpf"}]: dispatch 2024-03-04T16:15:08.235 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi083.hqmkpf"}]': finished 2024-03-04T16:15:08.235 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi086.yqigbl"}]: dispatch 2024-03-04T16:15:08.235 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi086.yqigbl"}]': finished 2024-03-04T16:15:08.235 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.235 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.235 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.235 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.235 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.235 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 170 B/s wr, 163 op/s 2024-03-04T16:15:08.236 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.236 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.236 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.236 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.236 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:15:08.236 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:15:08.236 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.236 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:15:08.236 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.236 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.236 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.236 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.236 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi083.szjzkw"}]: dispatch 2024-03-04T16:15:08.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi083.szjzkw"}]': finished 2024-03-04T16:15:08.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi086.ezumfo"}]: dispatch 2024-03-04T16:15:08.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi086.ezumfo"}]': finished 2024-03-04T16:15:08.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi083.hqmkpf"}]: dispatch 2024-03-04T16:15:08.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi083.hqmkpf"}]': finished 2024-03-04T16:15:08.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi086.yqigbl"}]: dispatch 2024-03-04T16:15:08.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi086.yqigbl"}]': finished 2024-03-04T16:15:08.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:07 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 170 B/s wr, 163 op/s 2024-03-04T16:15:08.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:15:08.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:15:08.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:15:08.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi083.szjzkw"}]: dispatch 2024-03-04T16:15:08.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi083.szjzkw"}]': finished 2024-03-04T16:15:08.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi086.ezumfo"}]: dispatch 2024-03-04T16:15:08.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi086.ezumfo"}]': finished 2024-03-04T16:15:08.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi083.hqmkpf"}]: dispatch 2024-03-04T16:15:08.240 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi083.hqmkpf"}]': finished 2024-03-04T16:15:08.240 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi086.yqigbl"}]: dispatch 2024-03-04T16:15:08.240 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi086.yqigbl"}]': finished 2024-03-04T16:15:08.240 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.240 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:08.240 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.240 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:08.240 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:07 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:09.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:08 smithi086 ceph-mon[57919]: Detected new or changed devices on smithi086 2024-03-04T16:15:09.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:08 smithi086 ceph-mon[57919]: Detected new or changed devices on smithi083 2024-03-04T16:15:09.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:08 smithi086 ceph-mon[57919]: Upgrade: Setting container_image for all rgw 2024-03-04T16:15:09.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:08 smithi086 ceph-mon[57919]: Upgrade: Setting container_image for all rbd-mirror 2024-03-04T16:15:09.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:08 smithi086 ceph-mon[57919]: Upgrade: Setting container_image for all ceph-exporter 2024-03-04T16:15:09.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:08 smithi083 ceph-mon[64844]: Detected new or changed devices on smithi086 2024-03-04T16:15:09.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:08 smithi083 ceph-mon[64844]: Detected new or changed devices on smithi083 2024-03-04T16:15:09.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:08 smithi083 ceph-mon[64844]: Upgrade: Setting container_image for all rgw 2024-03-04T16:15:09.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:08 smithi083 ceph-mon[64844]: Upgrade: Setting container_image for all rbd-mirror 2024-03-04T16:15:09.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:08 smithi083 ceph-mon[64844]: Upgrade: Setting container_image for all ceph-exporter 2024-03-04T16:15:09.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:08 smithi083 ceph-mon[67383]: Detected new or changed devices on smithi086 2024-03-04T16:15:09.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:08 smithi083 ceph-mon[67383]: Detected new or changed devices on smithi083 2024-03-04T16:15:09.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:08 smithi083 ceph-mon[67383]: Upgrade: Setting container_image for all rgw 2024-03-04T16:15:09.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:08 smithi083 ceph-mon[67383]: Upgrade: Setting container_image for all rbd-mirror 2024-03-04T16:15:09.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:08 smithi083 ceph-mon[67383]: Upgrade: Setting container_image for all ceph-exporter 2024-03-04T16:15:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:09 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:09 smithi086 ceph-mon[57919]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-04T16:15:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:09 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:09 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:15:10.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:09 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:15:10.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:09 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:10.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:09 smithi083 ceph-mon[64844]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-04T16:15:10.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:09 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:10.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:09 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:15:10.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:09 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:15:10.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:09 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:10.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:09 smithi083 ceph-mon[67383]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-04T16:15:10.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:09 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:10.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:09 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi083.fnypow", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:15:10.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:09 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:15:11.230 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:10 smithi083 ceph-mon[64844]: Upgrade: Updating iscsi.foo.smithi083.fnypow 2024-03-04T16:15:11.230 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:10 smithi083 ceph-mon[64844]: Deploying daemon iscsi.foo.smithi083.fnypow on smithi083 2024-03-04T16:15:11.230 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:10 smithi083 ceph-mon[67383]: Upgrade: Updating iscsi.foo.smithi083.fnypow 2024-03-04T16:15:11.230 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:10 smithi083 ceph-mon[67383]: Deploying daemon iscsi.foo.smithi083.fnypow on smithi083 2024-03-04T16:15:11.233 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:10 smithi086 ceph-mon[57919]: Upgrade: Updating iscsi.foo.smithi083.fnypow 2024-03-04T16:15:11.233 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:10 smithi086 ceph-mon[57919]: Deploying daemon iscsi.foo.smithi083.fnypow on smithi083 2024-03-04T16:15:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:11 smithi086 ceph-mon[57919]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-04T16:15:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:12.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:11 smithi083 ceph-mon[64844]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-04T16:15:12.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:12.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:11 smithi083 ceph-mon[67383]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-04T16:15:12.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:14.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:13 smithi086 ceph-mon[57919]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 85 B/s wr, 41 op/s 2024-03-04T16:15:14.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:13 smithi083 ceph-mon[64844]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 85 B/s wr, 41 op/s 2024-03-04T16:15:14.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:13 smithi083 ceph-mon[67383]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 85 B/s wr, 41 op/s 2024-03-04T16:15:15.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:15 smithi083 ceph-mon[64844]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-04T16:15:15.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:15:15 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:15:15] "GET /metrics HTTP/1.1" 200 38688 "" "Prometheus/2.33.4" 2024-03-04T16:15:15.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:15 smithi083 ceph-mon[67383]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-04T16:15:16.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:15 smithi086 ceph-mon[57919]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-04T16:15:18.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:17 smithi086 ceph-mon[57919]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:15:18.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:17 smithi083 ceph-mon[64844]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:15:18.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:17 smithi083 ceph-mon[67383]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:15:20.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:19 smithi086 ceph-mon[57919]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:20.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:19 smithi086 ceph-mon[57919]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:15:20.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:19 smithi083 ceph-mon[64844]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:20.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:19 smithi083 ceph-mon[64844]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:15:20.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:19 smithi083 ceph-mon[67383]: from='client.25315 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:20.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:19 smithi083 ceph-mon[67383]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:15:22.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:21 smithi086 ceph-mon[57919]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:15:22.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:21 smithi083 ceph-mon[64844]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:15:22.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:21 smithi083 ceph-mon[67383]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:15:22.977 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:15:22 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:15:22.661+0000 7fe833559700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi083.fnypow on host smithi083 failed. 2024-03-04T16:15:23.891 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:23 smithi083 ceph-mon[64844]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi083.fnypow on host smithi083 failed. 2024-03-04T16:15:23.892 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:23 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:23.892 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:23 smithi083 ceph-mon[64844]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:15:23.892 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:23 smithi083 ceph-mon[67383]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi083.fnypow on host smithi083 failed. 2024-03-04T16:15:23.892 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:23 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:23.892 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:23 smithi083 ceph-mon[67383]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:15:23.983 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:23 smithi086 ceph-mon[57919]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi083.fnypow on host smithi083 failed. 2024-03-04T16:15:23.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:23 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:23.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:23 smithi086 ceph-mon[57919]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:15:24.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:24 smithi086 ceph-mon[57919]: Health check failed: Upgrading daemon iscsi.foo.smithi083.fnypow on host smithi083 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-04T16:15:24.987 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:24 smithi083 ceph-mon[67383]: Health check failed: Upgrading daemon iscsi.foo.smithi083.fnypow on host smithi083 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-04T16:15:24.987 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:24 smithi083 ceph-mon[64844]: Health check failed: Upgrading daemon iscsi.foo.smithi083.fnypow on host smithi083 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-04T16:15:25.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:25 smithi086 ceph-mon[57919]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-03-04T16:15:25.987 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:15:25 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:15:25] "GET /metrics HTTP/1.1" 200 38688 "" "Prometheus/2.33.4" 2024-03-04T16:15:25.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:25 smithi083 ceph-mon[67383]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-03-04T16:15:25.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:25 smithi083 ceph-mon[64844]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-03-04T16:15:27.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:27.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:27.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:27.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:27.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:15:27.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:15:27.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:27 smithi086 ceph-mon[57919]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-03-04T16:15:27.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:27 smithi086 ceph-mon[57919]: Checking dashboard <-> RGW credentials 2024-03-04T16:15:27.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:27.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:27.485 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:27.487 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:27.487 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:15:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:15:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[67383]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-03-04T16:15:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[67383]: Checking dashboard <-> RGW credentials 2024-03-04T16:15:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:27.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:15:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:15:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[64844]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-03-04T16:15:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[64844]: Checking dashboard <-> RGW credentials 2024-03-04T16:15:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:27.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:28.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:28 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:15:28.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:28 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:15:28.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:28 smithi086 ceph-mon[57919]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 495 B/s rd, 0 op/s 2024-03-04T16:15:28.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:28 smithi086 ceph-mon[57919]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s 2024-03-04T16:15:28.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:28 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:28.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:28 smithi086 ceph-mon[57919]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-04T16:15:28.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:28 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:15:28.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:28 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:15:28.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:28 smithi083 ceph-mon[64844]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 495 B/s rd, 0 op/s 2024-03-04T16:15:28.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:28 smithi083 ceph-mon[64844]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s 2024-03-04T16:15:28.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:28 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:28.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:28 smithi083 ceph-mon[64844]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-04T16:15:28.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:28 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:15:28.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:28 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:15:28.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:28 smithi083 ceph-mon[67383]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 495 B/s rd, 0 op/s 2024-03-04T16:15:28.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:28 smithi083 ceph-mon[67383]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s 2024-03-04T16:15:28.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:28 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:28.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:28 smithi083 ceph-mon[67383]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-04T16:15:30.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:30 smithi086 ceph-mon[57919]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s 2024-03-04T16:15:30.737 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:30 smithi083 ceph-mon[64844]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s 2024-03-04T16:15:30.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:30 smithi083 ceph-mon[67383]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s 2024-03-04T16:15:32.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:32 smithi086 ceph-mon[57919]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-03-04T16:15:32.737 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:32 smithi083 ceph-mon[64844]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-03-04T16:15:32.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:32 smithi083 ceph-mon[67383]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-03-04T16:15:32.954 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:15:33.296 INFO:teuthology.orchestra.run.smithi083.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi083.fnypow on host smithi083 failed." 2024-03-04T16:15:33.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:33 smithi086 ceph-mon[57919]: from='client.54661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:33.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:33 smithi083 ceph-mon[67383]: from='client.54661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:33.737 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:33 smithi083 ceph-mon[64844]: from='client.54661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:33.952 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph orch ps' 2024-03-04T16:15:34.523 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:34 smithi083 ceph-mon[67383]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-03-04T16:15:34.523 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:34 smithi083 ceph-mon[67383]: from='client.54667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:34.523 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:34 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/2798598057' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T16:15:34.523 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:34 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/975017746' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1685120892"}]: dispatch 2024-03-04T16:15:34.524 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:34 smithi083 ceph-mon[64844]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-03-04T16:15:34.524 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:34 smithi083 ceph-mon[64844]: from='client.54667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:34.524 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:34 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/2798598057' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T16:15:34.524 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:34 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/975017746' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1685120892"}]: dispatch 2024-03-04T16:15:34.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:34 smithi086 ceph-mon[57919]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-03-04T16:15:34.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:34 smithi086 ceph-mon[57919]: from='client.54667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:34.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:34 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/2798598057' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T16:15:34.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:34 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/975017746' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1685120892"}]: dispatch 2024-03-04T16:15:35.562 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:35 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/975017746' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1685120892"}]': finished 2024-03-04T16:15:35.563 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:35 smithi083 ceph-mon[64844]: osdmap e148: 8 total, 8 up, 8 in 2024-03-04T16:15:35.563 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:35 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/3803119544' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2641666911"}]: dispatch 2024-03-04T16:15:35.563 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:35 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/975017746' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1685120892"}]': finished 2024-03-04T16:15:35.563 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:35 smithi083 ceph-mon[67383]: osdmap e148: 8 total, 8 up, 8 in 2024-03-04T16:15:35.563 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:35 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/3803119544' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2641666911"}]: dispatch 2024-03-04T16:15:35.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:35 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/975017746' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/1685120892"}]': finished 2024-03-04T16:15:35.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:35 smithi086 ceph-mon[57919]: osdmap e148: 8 total, 8 up, 8 in 2024-03-04T16:15:35.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:35 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/3803119544' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2641666911"}]: dispatch 2024-03-04T16:15:35.987 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:15:35 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:15:35] "GET /metrics HTTP/1.1" 200 38766 "" "Prometheus/2.33.4" 2024-03-04T16:15:36.007 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:15:36.008 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (8m) 9s ago 13m 24.3M - 0.23.0 ba2b418f427c 3c6dc3779a0e 2024-03-04T16:15:36.008 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (7m) 9s ago 13m 54.2M - 8.3.5 dad864ee21e9 fb754d58c53f 2024-03-04T16:15:36.008 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 unknown 9s ago 12m - - 2024-03-04T16:15:36.008 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443,9283,8765 running (7m) 9s ago 17m 452M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c33100cb8a8 2024-03-04T16:15:36.008 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:8443,9283,8765 running (7m) 9s ago 19m 513M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 164dd93609ee 2024-03-04T16:15:36.008 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (6m) 9s ago 19m 58.9M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c173a2442197 2024-03-04T16:15:36.008 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (6m) 9s ago 18m 44.0M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 71557985ed5a 2024-03-04T16:15:36.009 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (6m) 9s ago 18m 47.9M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e eff1982ba275 2024-03-04T16:15:36.009 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (8m) 9s ago 14m 17.9M - 1.3.1 1dbe0e931976 9bf54b460640 2024-03-04T16:15:36.009 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (8m) 9s ago 14m 15.6M - 1.3.1 1dbe0e931976 b72dd80b2588 2024-03-04T16:15:36.009 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (5m) 9s ago 17m 53.8M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e f7ecdc5b1520 2024-03-04T16:15:36.009 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (4m) 9s ago 17m 58.5M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 2c648d9a2b38 2024-03-04T16:15:36.009 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (4m) 9s ago 16m 61.7M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c971c1f3f0a 2024-03-04T16:15:36.009 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (3m) 9s ago 16m 54.3M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 8991d467d41c 2024-03-04T16:15:36.009 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (2m) 9s ago 16m 63.0M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 912383028c72 2024-03-04T16:15:36.009 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (2m) 9s ago 15m 57.6M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 53215200f312 2024-03-04T16:15:36.009 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (105s) 9s ago 15m 46.7M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e a4584e5b3044 2024-03-04T16:15:36.010 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (70s) 9s ago 14m 59.3M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 881253b7c5f1 2024-03-04T16:15:36.010 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 running (7m) 9s ago 14m 71.6M - 2.33.4 514e6a882f6e b563a246eeac 2024-03-04T16:15:36.010 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (54s) 9s ago 12m 75.1M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 51a6984123f3 2024-03-04T16:15:36.010 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (43s) 9s ago 12m 76.3M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 2e29f91b67f8 2024-03-04T16:15:36.010 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (48s) 9s ago 12m 75.4M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4ea5fe6746fd 2024-03-04T16:15:36.010 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (37s) 9s ago 12m 77.6M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e cea9400093b8 2024-03-04T16:15:36.701 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:36 smithi083 ceph-mon[64844]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-04T16:15:36.701 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:36 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/3803119544' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2641666911"}]': finished 2024-03-04T16:15:36.701 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:36 smithi083 ceph-mon[64844]: osdmap e149: 8 total, 8 up, 8 in 2024-03-04T16:15:36.701 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:36 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/4057677018' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/3357838142"}]: dispatch 2024-03-04T16:15:36.701 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:36 smithi083 ceph-mon[64844]: from='client.54715 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:36.701 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:36 smithi083 ceph-mon[67383]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-04T16:15:36.702 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:36 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/3803119544' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2641666911"}]': finished 2024-03-04T16:15:36.702 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:36 smithi083 ceph-mon[67383]: osdmap e149: 8 total, 8 up, 8 in 2024-03-04T16:15:36.702 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:36 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/4057677018' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/3357838142"}]: dispatch 2024-03-04T16:15:36.702 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:36 smithi083 ceph-mon[67383]: from='client.54715 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:36.722 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph versions' 2024-03-04T16:15:36.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:36 smithi086 ceph-mon[57919]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-04T16:15:36.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:36 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/3803119544' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2641666911"}]': finished 2024-03-04T16:15:36.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:36 smithi086 ceph-mon[57919]: osdmap e149: 8 total, 8 up, 8 in 2024-03-04T16:15:36.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:36 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/4057677018' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/3357838142"}]: dispatch 2024-03-04T16:15:36.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:36 smithi086 ceph-mon[57919]: from='client.54715 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:15:37.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:37 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/4057677018' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/3357838142"}]': finished 2024-03-04T16:15:37.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:37 smithi086 ceph-mon[57919]: osdmap e150: 8 total, 8 up, 8 in 2024-03-04T16:15:37.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:37 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/263129916' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/479021090"}]: dispatch 2024-03-04T16:15:37.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:37 smithi086 ceph-mon[57919]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/479021090"}]: dispatch 2024-03-04T16:15:37.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:37 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/4057677018' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/3357838142"}]': finished 2024-03-04T16:15:37.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:37 smithi083 ceph-mon[67383]: osdmap e150: 8 total, 8 up, 8 in 2024-03-04T16:15:37.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:37 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/263129916' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/479021090"}]: dispatch 2024-03-04T16:15:37.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:37 smithi083 ceph-mon[67383]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/479021090"}]: dispatch 2024-03-04T16:15:37.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:37 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/4057677018' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/3357838142"}]': finished 2024-03-04T16:15:37.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:37 smithi083 ceph-mon[64844]: osdmap e150: 8 total, 8 up, 8 in 2024-03-04T16:15:37.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:37 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/263129916' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/479021090"}]: dispatch 2024-03-04T16:15:37.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:37 smithi083 ceph-mon[64844]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/479021090"}]: dispatch 2024-03-04T16:15:38.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:38 smithi086 ceph-mon[57919]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:15:38.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:38 smithi086 ceph-mon[57919]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/479021090"}]': finished 2024-03-04T16:15:38.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:38 smithi086 ceph-mon[57919]: osdmap e151: 8 total, 8 up, 8 in 2024-03-04T16:15:38.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:38 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/2642574412' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/479021090"}]: dispatch 2024-03-04T16:15:38.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:38 smithi083 ceph-mon[64844]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:15:38.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:38 smithi083 ceph-mon[64844]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/479021090"}]': finished 2024-03-04T16:15:38.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:38 smithi083 ceph-mon[64844]: osdmap e151: 8 total, 8 up, 8 in 2024-03-04T16:15:38.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:38 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/2642574412' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/479021090"}]: dispatch 2024-03-04T16:15:38.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:38 smithi083 ceph-mon[67383]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:15:38.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:38 smithi083 ceph-mon[67383]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6801/479021090"}]': finished 2024-03-04T16:15:38.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:38 smithi083 ceph-mon[67383]: osdmap e151: 8 total, 8 up, 8 in 2024-03-04T16:15:38.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:38 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/2642574412' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/479021090"}]: dispatch 2024-03-04T16:15:38.810 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:15:38.810 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:15:38.810 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 3 2024-03-04T16:15:38.810 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:15:38.811 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:15:38.811 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:15:38.811 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:15:38.811 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:15:38.811 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 8 2024-03-04T16:15:38.811 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:15:38.811 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:15:38.811 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 4 2024-03-04T16:15:38.811 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:15:38.811 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:15:38.811 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 17 2024-03-04T16:15:38.811 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:15:38.811 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:15:39.475 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:39 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/2642574412' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/479021090"}]': finished 2024-03-04T16:15:39.475 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:39 smithi083 ceph-mon[64844]: osdmap e152: 8 total, 8 up, 8 in 2024-03-04T16:15:39.475 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:39 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/2383621862' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2304521942"}]: dispatch 2024-03-04T16:15:39.476 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:39 smithi083 ceph-mon[64844]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2304521942"}]: dispatch 2024-03-04T16:15:39.476 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:39 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/562276347' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:39.476 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:39 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/2642574412' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/479021090"}]': finished 2024-03-04T16:15:39.476 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:39 smithi083 ceph-mon[67383]: osdmap e152: 8 total, 8 up, 8 in 2024-03-04T16:15:39.476 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:39 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/2383621862' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2304521942"}]: dispatch 2024-03-04T16:15:39.476 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:39 smithi083 ceph-mon[67383]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2304521942"}]: dispatch 2024-03-04T16:15:39.476 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:39 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/562276347' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:39.500 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-04T16:15:39.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:39 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/2642574412' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:6800/479021090"}]': finished 2024-03-04T16:15:39.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:39 smithi086 ceph-mon[57919]: osdmap e152: 8 total, 8 up, 8 in 2024-03-04T16:15:39.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:39 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/2383621862' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2304521942"}]: dispatch 2024-03-04T16:15:39.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:39 smithi086 ceph-mon[57919]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2304521942"}]: dispatch 2024-03-04T16:15:39.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:39 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/562276347' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:15:40.466 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:40 smithi083 ceph-mon[64844]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-04T16:15:40.466 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:40 smithi083 ceph-mon[64844]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2304521942"}]': finished 2024-03-04T16:15:40.466 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:40 smithi083 ceph-mon[64844]: osdmap e153: 8 total, 8 up, 8 in 2024-03-04T16:15:40.466 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:40 smithi083 ceph-mon[67383]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-04T16:15:40.466 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:40 smithi083 ceph-mon[67383]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2304521942"}]': finished 2024-03-04T16:15:40.467 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:40 smithi083 ceph-mon[67383]: osdmap e153: 8 total, 8 up, 8 in 2024-03-04T16:15:40.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:40 smithi086 ceph-mon[57919]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-04T16:15:40.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:40 smithi086 ceph-mon[57919]: from='client.? ' entity='client.iscsi.foo.smithi083.fnypow' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.83:0/2304521942"}]': finished 2024-03-04T16:15:40.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:40 smithi086 ceph-mon[57919]: osdmap e153: 8 total, 8 up, 8 in 2024-03-04T16:15:41.361 INFO:teuthology.orchestra.run.smithi083.stdout:wait for servicemap items w/ changing names to refresh 2024-03-04T16:15:41.987 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'sleep 60' 2024-03-04T16:15:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:42 smithi086 ceph-mon[57919]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-04T16:15:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:42 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:42 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:43.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:42 smithi083 ceph-mon[67383]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-04T16:15:43.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:42 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:43.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:42 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:43.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:42 smithi083 ceph-mon[64844]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-04T16:15:43.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:42 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:15:43.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:42 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:45.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:44 smithi086 ceph-mon[57919]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:15:45.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:44 smithi086 ceph-mon[57919]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:45.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:44 smithi083 ceph-mon[64844]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:15:45.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:44 smithi083 ceph-mon[64844]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:45.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:44 smithi083 ceph-mon[67383]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:15:45.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:44 smithi083 ceph-mon[67383]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:45.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:15:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:15:45] "GET /metrics HTTP/1.1" 200 38763 "" "Prometheus/2.33.4" 2024-03-04T16:15:47.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:46 smithi086 ceph-mon[57919]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 646 B/s rd, 0 op/s 2024-03-04T16:15:47.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:46 smithi083 ceph-mon[64844]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 646 B/s rd, 0 op/s 2024-03-04T16:15:47.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:46 smithi083 ceph-mon[67383]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 646 B/s rd, 0 op/s 2024-03-04T16:15:49.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:48 smithi086 ceph-mon[57919]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:15:49.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:48 smithi083 ceph-mon[64844]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:15:49.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:48 smithi083 ceph-mon[67383]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:15:51.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:50 smithi086 ceph-mon[57919]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:15:51.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:50 smithi083 ceph-mon[64844]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:15:51.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:50 smithi083 ceph-mon[67383]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:15:53.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:52 smithi086 ceph-mon[57919]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:15:53.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:52 smithi083 ceph-mon[64844]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:15:53.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:52 smithi083 ceph-mon[67383]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:15:54.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:54 smithi083 ceph-mon[67383]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:15:54.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:54 smithi083 ceph-mon[67383]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:54.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:54 smithi083 ceph-mon[64844]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:15:54.989 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:54 smithi083 ceph-mon[64844]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:55.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:54 smithi086 ceph-mon[57919]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:15:55.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:54 smithi086 ceph-mon[57919]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:55.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:15:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:15:55] "GET /metrics HTTP/1.1" 200 38763 "" "Prometheus/2.33.4" 2024-03-04T16:15:57.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:56 smithi086 ceph-mon[57919]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:15:57.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:56 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:57.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:56 smithi083 ceph-mon[67383]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:15:57.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:56 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:57.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:56 smithi083 ceph-mon[64844]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:15:57.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:56 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:59.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:15:58 smithi086 ceph-mon[57919]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:15:59.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:15:58 smithi083 ceph-mon[64844]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:15:59.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:15:58 smithi083 ceph-mon[67383]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:01.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:00 smithi086 ceph-mon[57919]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:01.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:00 smithi083 ceph-mon[64844]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:01.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:00 smithi083 ceph-mon[67383]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:03.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:02 smithi086 ceph-mon[57919]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:03.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:02 smithi083 ceph-mon[67383]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:03.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:02 smithi083 ceph-mon[64844]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:05.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:04 smithi086 ceph-mon[57919]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:05.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:04 smithi086 ceph-mon[57919]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:05.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:04 smithi083 ceph-mon[67383]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:05.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:04 smithi083 ceph-mon[67383]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:05.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:04 smithi083 ceph-mon[64844]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:05.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:04 smithi083 ceph-mon[64844]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:05.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:16:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:16:05] "GET /metrics HTTP/1.1" 200 38763 "" "Prometheus/2.33.4" 2024-03-04T16:16:07.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:06 smithi086 ceph-mon[57919]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:07.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:06 smithi083 ceph-mon[64844]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:07.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:06 smithi083 ceph-mon[67383]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:09.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:08 smithi086 ceph-mon[57919]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:09.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:08 smithi083 ceph-mon[67383]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:09.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:08 smithi083 ceph-mon[64844]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:11.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:10 smithi086 ceph-mon[57919]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:11.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:10 smithi083 ceph-mon[64844]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:11.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:10 smithi083 ceph-mon[67383]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:12.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:12.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:13.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:12 smithi086 ceph-mon[57919]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:13.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:12 smithi083 ceph-mon[67383]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:13.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:12 smithi083 ceph-mon[64844]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:15.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:14 smithi086 ceph-mon[57919]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:15.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:14 smithi086 ceph-mon[57919]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:15.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:14 smithi083 ceph-mon[67383]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:15.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:14 smithi083 ceph-mon[67383]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:15.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:14 smithi083 ceph-mon[64844]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:15.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:14 smithi083 ceph-mon[64844]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:15.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:16:15 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:16:15] "GET /metrics HTTP/1.1" 200 38762 "" "Prometheus/2.33.4" 2024-03-04T16:16:17.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:16 smithi086 ceph-mon[57919]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:17.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:16 smithi083 ceph-mon[67383]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:17.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:16 smithi083 ceph-mon[64844]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:19.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:18 smithi086 ceph-mon[57919]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:19.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:18 smithi083 ceph-mon[64844]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:19.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:18 smithi083 ceph-mon[67383]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:21.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:20 smithi086 ceph-mon[57919]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:21.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:20 smithi083 ceph-mon[64844]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:21.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:20 smithi083 ceph-mon[67383]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:23.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:22 smithi086 ceph-mon[57919]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:23.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:22 smithi083 ceph-mon[67383]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:23.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:22 smithi083 ceph-mon[64844]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:25.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:24 smithi083 ceph-mon[67383]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:25.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:24 smithi083 ceph-mon[67383]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:25.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:24 smithi083 ceph-mon[64844]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:25.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:24 smithi083 ceph-mon[64844]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:25.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:24 smithi086 ceph-mon[57919]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:25.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:24 smithi086 ceph-mon[57919]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:25.987 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:16:25 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:16:25] "GET /metrics HTTP/1.1" 200 38762 "" "Prometheus/2.33.4" 2024-03-04T16:16:27.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:26 smithi083 ceph-mon[67383]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:27.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:26 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:27.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:26 smithi083 ceph-mon[64844]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:27.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:26 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:27.286 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:26 smithi086 ceph-mon[57919]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:27.286 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:26 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:28.483 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:16:28.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:16:28.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:27 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:16:28.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:16:28.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:16:28.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:27 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:16:28.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:16:28.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:16:28.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:27 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' 2024-03-04T16:16:29.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:28 smithi086 ceph-mon[57919]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:29.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:28 smithi086 ceph-mon[57919]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 956 B/s rd, 0 op/s 2024-03-04T16:16:29.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:28 smithi086 ceph-mon[57919]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:16:29.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:28 smithi083 ceph-mon[64844]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:29.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:28 smithi083 ceph-mon[64844]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 956 B/s rd, 0 op/s 2024-03-04T16:16:29.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:28 smithi083 ceph-mon[64844]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:16:29.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:28 smithi083 ceph-mon[67383]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:29.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:28 smithi083 ceph-mon[67383]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 956 B/s rd, 0 op/s 2024-03-04T16:16:29.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:28 smithi083 ceph-mon[67383]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:16:31.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:30 smithi086 ceph-mon[57919]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-04T16:16:31.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:30 smithi083 ceph-mon[64844]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-04T16:16:31.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:30 smithi083 ceph-mon[67383]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-04T16:16:33.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:33 smithi086 ceph-mon[57919]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:16:33.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:33 smithi083 ceph-mon[64844]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:16:33.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:33 smithi083 ceph-mon[67383]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:16:35.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:35 smithi086 ceph-mon[57919]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:35.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:35 smithi086 ceph-mon[57919]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:16:35.487 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:35 smithi083 ceph-mon[67383]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:35.487 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:35 smithi083 ceph-mon[67383]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:16:35.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:35 smithi083 ceph-mon[64844]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:35.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:35 smithi083 ceph-mon[64844]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:16:35.987 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:16:35 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:16:35] "GET /metrics HTTP/1.1" 200 38766 "" "Prometheus/2.33.4" 2024-03-04T16:16:37.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:37 smithi086 ceph-mon[57919]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-04T16:16:37.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:37 smithi083 ceph-mon[64844]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-04T16:16:37.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:37 smithi083 ceph-mon[67383]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-04T16:16:39.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:39 smithi086 ceph-mon[57919]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:16:39.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:39 smithi083 ceph-mon[64844]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:16:39.487 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:39 smithi083 ceph-mon[67383]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:16:41.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:41 smithi086 ceph-mon[57919]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:41.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:41 smithi083 ceph-mon[64844]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:41.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:41 smithi083 ceph-mon[67383]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:42.389 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:42 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:42.389 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:42 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:42.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:42 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:43.414 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:43 smithi083 ceph-mon[64844]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:43.415 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:43 smithi083 ceph-mon[67383]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:43.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:43 smithi086 ceph-mon[57919]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:44.376 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph orch ps' 2024-03-04T16:16:45.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:45 smithi083 ceph-mon[67383]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:45.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:45 smithi083 ceph-mon[67383]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:45.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:45 smithi083 ceph-mon[64844]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:45.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:45 smithi083 ceph-mon[64844]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:45.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:45 smithi086 ceph-mon[57919]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:45.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:45 smithi086 ceph-mon[57919]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:45.977 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:16:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:16:45] "GET /metrics HTTP/1.1" 200 38760 "" "Prometheus/2.33.4" 2024-03-04T16:16:46.415 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T16:16:46.415 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.a smithi083 *:9093,9094 running (9m) 79s ago 14m 24.3M - 0.23.0 ba2b418f427c 3c6dc3779a0e 2024-03-04T16:16:46.415 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.a smithi086 *:3000 running (8m) 80s ago 14m 54.2M - 8.3.5 dad864ee21e9 fb754d58c53f 2024-03-04T16:16:46.415 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi.foo.smithi083.fnypow smithi083 unknown 79s ago 13m - - 2024-03-04T16:16:46.415 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.x smithi086 *:8443,9283,8765 running (8m) 80s ago 19m 452M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c33100cb8a8 2024-03-04T16:16:46.415 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.y smithi083 *:8443,9283,8765 running (8m) 79s ago 20m 513M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 164dd93609ee 2024-03-04T16:16:46.415 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a smithi083 running (7m) 79s ago 20m 58.9M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c173a2442197 2024-03-04T16:16:46.416 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b smithi086 running (7m) 80s ago 19m 44.0M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 71557985ed5a 2024-03-04T16:16:46.416 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c smithi083 running (7m) 79s ago 19m 47.9M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e eff1982ba275 2024-03-04T16:16:46.416 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.a smithi083 *:9100 running (9m) 79s ago 15m 17.9M - 1.3.1 1dbe0e931976 9bf54b460640 2024-03-04T16:16:46.416 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.b smithi086 *:9100 running (9m) 80s ago 15m 15.6M - 1.3.1 1dbe0e931976 b72dd80b2588 2024-03-04T16:16:46.416 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (6m) 79s ago 18m 53.8M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e f7ecdc5b1520 2024-03-04T16:16:46.416 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (5m) 79s ago 18m 58.5M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 2c648d9a2b38 2024-03-04T16:16:46.416 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (5m) 79s ago 18m 61.7M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c971c1f3f0a 2024-03-04T16:16:46.416 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (4m) 79s ago 17m 54.3M 2736M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 8991d467d41c 2024-03-04T16:16:46.416 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi086 running (4m) 80s ago 17m 63.0M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 912383028c72 2024-03-04T16:16:46.416 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi086 running (3m) 80s ago 16m 57.6M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 53215200f312 2024-03-04T16:16:46.416 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi086 running (2m) 80s ago 16m 46.7M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e a4584e5b3044 2024-03-04T16:16:46.416 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi086 running (2m) 80s ago 16m 59.3M 2992M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 881253b7c5f1 2024-03-04T16:16:46.416 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.a smithi086 *:9095 running (8m) 80s ago 15m 71.6M - 2.33.4 514e6a882f6e b563a246eeac 2024-03-04T16:16:46.416 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi083.szjzkw smithi083 *:8000 running (2m) 79s ago 13m 75.1M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 51a6984123f3 2024-03-04T16:16:46.417 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.foo.smithi086.ezumfo smithi086 *:8000 running (113s) 80s ago 13m 76.3M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 2e29f91b67f8 2024-03-04T16:16:46.417 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi083.hqmkpf smithi083 *:80 running (119s) 79s ago 13m 75.4M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4ea5fe6746fd 2024-03-04T16:16:46.417 INFO:teuthology.orchestra.run.smithi083.stdout:rgw.smpl.smithi086.yqigbl smithi086 *:80 running (107s) 80s ago 13m 77.6M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e cea9400093b8 2024-03-04T16:16:47.116 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph versions' 2024-03-04T16:16:47.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:47 smithi083 ceph-mon[67383]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:47.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:47 smithi083 ceph-mon[64844]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:47.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:47 smithi086 ceph-mon[57919]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:48.424 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:48 smithi083 ceph-mon[67383]: from='client.54745 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:16:48.424 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:48 smithi083 ceph-mon[64844]: from='client.54745 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:16:48.483 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:48 smithi086 ceph-mon[57919]: from='client.54745 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:16:49.173 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:16:49.173 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-03-04T16:16:49.173 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 3 2024-03-04T16:16:49.173 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:16:49.174 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-03-04T16:16:49.174 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 2 2024-03-04T16:16:49.174 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:16:49.174 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-03-04T16:16:49.174 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 8 2024-03-04T16:16:49.174 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:16:49.174 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": { 2024-03-04T16:16:49.174 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 4 2024-03-04T16:16:49.174 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-04T16:16:49.174 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-03-04T16:16:49.174 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)": 17 2024-03-04T16:16:49.174 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-04T16:16:49.174 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:16:49.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:49 smithi086 ceph-mon[57919]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:49.487 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:49 smithi083 ceph-mon[67383]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:49.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:49 smithi083 ceph-mon[64844]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:49.931 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph orch upgrade status' 2024-03-04T16:16:50.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:50 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/2396698266' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:16:50.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:50 smithi083 ceph-mon[64844]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:50.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:50 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/2396698266' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:16:50.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:50 smithi083 ceph-mon[67383]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:50.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:50 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/2396698266' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:16:50.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:50 smithi086 ceph-mon[57919]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:51.967 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:16:51.967 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282", 2024-03-04T16:16:51.967 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-03-04T16:16:51.967 INFO:teuthology.orchestra.run.smithi083.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-04T16:16:51.967 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [ 2024-03-04T16:16:51.967 INFO:teuthology.orchestra.run.smithi083.stdout: "osd", 2024-03-04T16:16:51.967 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw", 2024-03-04T16:16:51.968 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr", 2024-03-04T16:16:51.968 INFO:teuthology.orchestra.run.smithi083.stdout: "mon" 2024-03-04T16:16:51.968 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-04T16:16:51.968 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "17/23 daemons upgraded", 2024-03-04T16:16:51.968 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi083.fnypow on host smithi083 failed.", 2024-03-04T16:16:51.968 INFO:teuthology.orchestra.run.smithi083.stdout: "is_paused": true 2024-03-04T16:16:51.968 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:16:52.585 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph health detail' 2024-03-04T16:16:53.225 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:53 smithi083 ceph-mon[64844]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:53.225 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:53 smithi083 ceph-mon[64844]: from='client.54757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:16:53.225 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:53 smithi083 ceph-mon[67383]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:53.225 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:53 smithi083 ceph-mon[67383]: from='client.54757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:16:53.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:53 smithi086 ceph-mon[57919]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:53.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:53 smithi086 ceph-mon[57919]: from='client.54757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:16:54.609 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi083.fnypow on host smithi083 failed. 2024-03-04T16:16:54.609 INFO:teuthology.orchestra.run.smithi083.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-04T16:16:54.609 INFO:teuthology.orchestra.run.smithi083.stdout: daemon iscsi.foo.smithi083.fnypow on smithi083 is in unknown state 2024-03-04T16:16:54.609 INFO:teuthology.orchestra.run.smithi083.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi083.fnypow on host smithi083 failed. 2024-03-04T16:16:54.609 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrade daemon: iscsi.foo.smithi083.fnypow: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi083.fnypow ... 2024-03-04T16:16:54.609 INFO:teuthology.orchestra.run.smithi083.stdout:Creating ceph-iscsi config... 2024-03-04T16:16:54.609 INFO:teuthology.orchestra.run.smithi083.stdout:Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/iscsi-gateway.cfg 2024-03-04T16:16:54.609 INFO:teuthology.orchestra.run.smithi083.stdout:Write file: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/iscsi.foo.smithi083.fnypow/tcmu-runner-entrypoint.sh 2024-03-04T16:16:54.609 INFO:teuthology.orchestra.run.smithi083.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d98ffad3c\x2dda3f\x2d11ee\x2d95c6\x2d87774f69a715.slice/ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service 2024-03-04T16:16:54.609 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 1 from systemctl start ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow 2024-03-04T16:16:54.610 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:16:54.610 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:16:54.610 INFO:teuthology.orchestra.run.smithi083.stdout:Traceback (most recent call last): 2024-03-04T16:16:54.610 INFO:teuthology.orchestra.run.smithi083.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-04T16:16:54.610 INFO:teuthology.orchestra.run.smithi083.stdout: return _run_code(code, main_globals, None, 2024-03-04T16:16:54.610 INFO:teuthology.orchestra.run.smithi083.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-04T16:16:54.610 INFO:teuthology.orchestra.run.smithi083.stdout: exec(code, run_globals) 2024-03-04T16:16:54.610 INFO:teuthology.orchestra.run.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5527, in 2024-03-04T16:16:54.610 INFO:teuthology.orchestra.run.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 5515, in main 2024-03-04T16:16:54.610 INFO:teuthology.orchestra.run.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3043, in command_deploy_from 2024-03-04T16:16:54.610 INFO:teuthology.orchestra.run.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3078, in _common_deploy 2024-03-04T16:16:54.610 INFO:teuthology.orchestra.run.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 3098, in _deploy_daemon_container 2024-03-04T16:16:54.610 INFO:teuthology.orchestra.run.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 1034, in deploy_daemon 2024-03-04T16:16:54.610 INFO:teuthology.orchestra.run.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/__main__.py", line 1170, in deploy_daemon_units 2024-03-04T16:16:54.611 INFO:teuthology.orchestra.run.smithi083.stdout: File "/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/cephadm.6bddf7e692133314beaeb23e32f4fc579a54b3daf6231939c1dfd5533c5c4b79/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-04T16:16:54.611 INFO:teuthology.orchestra.run.smithi083.stdout:RuntimeError: Failed command: systemctl start ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow: Job for ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service failed because the control process exited with error code. 2024-03-04T16:16:54.611 INFO:teuthology.orchestra.run.smithi083.stdout:See "systemctl status ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" and "journalctl -xeu ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@iscsi.foo.smithi083.fnypow.service" for details. 2024-03-04T16:16:55.227 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:55 smithi083 ceph-mon[67383]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:55.227 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:55 smithi083 ceph-mon[67383]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:55.227 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:55 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/3126506273' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:16:55.228 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:55 smithi083 ceph-mon[64844]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:55.228 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:55 smithi083 ceph-mon[64844]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:55.228 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:55 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/3126506273' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:16:55.254 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-04T16:16:55.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:55 smithi086 ceph-mon[57919]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:55.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:55 smithi086 ceph-mon[57919]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:55.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:55 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/3126506273' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-04T16:16:55.813 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:16:55 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:16:55] "GET /metrics HTTP/1.1" 200 38760 "" "Prometheus/2.33.4" 2024-03-04T16:16:57.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:57 smithi083 ceph-mon[67383]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:57.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:57 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:57.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:57 smithi083 ceph-mon[64844]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:57.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:57 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:57.285 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:57 smithi086 ceph-mon[57919]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:57.286 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:57 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:57.385 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-03-04T16:16:58.124 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-04T16:16:58.433 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:58 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/3733845633' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:16:58.433 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:58 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/3733845633' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:16:58.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:58 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/3733845633' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:16:59.454 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:16:59 smithi083 ceph-mon[64844]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:59.455 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:16:59 smithi083 ceph-mon[67383]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:59.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:16:59 smithi086 ceph-mon[57919]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:17:00.258 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)" 2024-03-04T16:17:00.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:00 smithi086 ceph-mon[57919]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:00.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:00 smithi083 ceph-mon[64844]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:00.487 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:00 smithi083 ceph-mon[67383]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:00.982 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -e sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-04T16:17:01.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:01 smithi083 ceph-mon[67383]: from='client.? 172.21.15.83:0/1514974229' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:17:01.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:01 smithi083 ceph-mon[64844]: from='client.? 172.21.15.83:0/1514974229' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:17:01.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:01 smithi086 ceph-mon[57919]: from='client.? 172.21.15.83:0/1514974229' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-04T16:17:02.460 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:02 smithi083 ceph-mon[64844]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:17:02.460 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:02 smithi083 ceph-mon[67383]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:17:02.483 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:02 smithi086 ceph-mon[57919]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:17:03.247 INFO:teuthology.orchestra.run.smithi083.stdout:osd 8 97s ago - 2024-03-04T16:17:03.839 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-04T16:17:03.850 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi083.front.sepia.ceph.com 2024-03-04T16:17:03.850 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- bash -c 'ceph orch upgrade ls' 2024-03-04T16:17:05.314 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:05 smithi083 ceph-mon[67383]: from='client.44856 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:17:05.314 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:05 smithi083 ceph-mon[67383]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:05.314 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:05 smithi083 ceph-mon[67383]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:05.314 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:05 smithi083 ceph-mon[64844]: from='client.44856 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:17:05.314 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:05 smithi083 ceph-mon[64844]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:05.314 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:05 smithi083 ceph-mon[64844]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:05.483 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:05 smithi086 ceph-mon[57919]: from='client.44856 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:17:05.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:05 smithi086 ceph-mon[57919]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:05.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:05 smithi086 ceph-mon[57919]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:05.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:17:05 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:17:05] "GET /metrics HTTP/1.1" 200 38762 "" "Prometheus/2.33.4" 2024-03-04T16:17:07.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:07 smithi086 ceph-mon[57919]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:07.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:07 smithi083 ceph-mon[64844]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:07.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:07 smithi083 ceph-mon[67383]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:08.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:08 smithi086 ceph-mon[57919]: from='client.54787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:17:08.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:08 smithi083 ceph-mon[64844]: from='client.54787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:17:08.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:08 smithi083 ceph-mon[67383]: from='client.54787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:17:08.507 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-04T16:17:08.507 INFO:teuthology.orchestra.run.smithi083.stdout: "image": "quay.io/ceph/ceph", 2024-03-04T16:17:08.507 INFO:teuthology.orchestra.run.smithi083.stdout: "registry": "quay.io", 2024-03-04T16:17:08.508 INFO:teuthology.orchestra.run.smithi083.stdout: "bare_image": "ceph/ceph", 2024-03-04T16:17:08.508 INFO:teuthology.orchestra.run.smithi083.stdout: "versions": [] 2024-03-04T16:17:08.508 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-04T16:17:09.208 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-03-04T16:17:09.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:09 smithi086 ceph-mon[57919]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:17:09.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:09 smithi083 ceph-mon[64844]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:17:09.487 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:09 smithi083 ceph-mon[67383]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:17:10.172 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:10 smithi083 ceph-mon[64844]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:10.173 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:10 smithi083 ceph-mon[67383]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:10.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:10 smithi086 ceph-mon[57919]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:12.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:11 smithi086 ceph-mon[57919]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:12.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:11 smithi083 ceph-mon[67383]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:12.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:11 smithi083 ceph-mon[64844]: from='mgr.25369 172.21.15.83:0/3338449836' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:13.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:12 smithi086 ceph-mon[57919]: from='client.54793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:17:13.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:12 smithi086 ceph-mon[57919]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:17:13.237 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:12 smithi083 ceph-mon[67383]: from='client.54793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:17:13.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:12 smithi083 ceph-mon[67383]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:17:13.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:12 smithi083 ceph-mon[64844]: from='client.54793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:17:13.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:12 smithi083 ceph-mon[64844]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:17:15.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:15 smithi086 ceph-mon[57919]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:15.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:15 smithi086 ceph-mon[57919]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:15.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:15 smithi083 ceph-mon[64844]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:15.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:15 smithi083 ceph-mon[64844]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:15.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:15 smithi083 ceph-mon[67383]: from='client.64520 -' entity='client.iscsi.foo.smithi083.fnypow' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:15.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:15 smithi083 ceph-mon[67383]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:15.987 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:17:15 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:17:15] "GET /metrics HTTP/1.1" 200 38764 "" "Prometheus/2.33.4" 2024-03-04T16:17:16.334 INFO:teuthology.orchestra.run.smithi083.stdout: "16.2.0", 2024-03-04T16:17:17.081 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-03-04T16:17:17.392 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:17 smithi083 ceph-mon[67383]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:17.392 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:17 smithi083 ceph-mon[64844]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:17.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:17 smithi086 ceph-mon[57919]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:19.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:19 smithi086 ceph-mon[57919]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:17:19.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:19 smithi083 ceph-mon[67383]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:17:19.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:19 smithi083 ceph-mon[64844]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:17:20.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:20 smithi086 ceph-mon[57919]: from='client.44874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:17:20.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:20 smithi086 ceph-mon[57919]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:20.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:20 smithi083 ceph-mon[64844]: from='client.44874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:17:20.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:20 smithi083 ceph-mon[64844]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:20.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:20 smithi083 ceph-mon[67383]: from='client.44874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:17:20.488 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:20 smithi083 ceph-mon[67383]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:17:22.427 INFO:teuthology.orchestra.run.smithi083.stdout: "v16.2.2", 2024-03-04T16:17:22.427 INFO:teuthology.orchestra.run.smithi083.stdout: "v16.2.2-20210505", 2024-03-04T16:17:23.101 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:23 smithi083 ceph-mon[67383]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:17:23.102 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:23 smithi083 ceph-mon[64844]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:17:23.128 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-04T16:17:23.139 INFO:tasks.cephadm:Teardown begin 2024-03-04T16:17:23.139 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T16:17:23.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T16:17:23.207 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-04T16:17:23.207 DEBUG:teuthology.orchestra.run.smithi083:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-04T16:17:23.228 DEBUG:teuthology.orchestra.run.smithi086:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-04T16:17:23.264 INFO:tasks.cephadm:Stopping all daemons... 2024-03-04T16:17:23.265 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-04T16:17:23.265 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mon.a 2024-03-04T16:17:23.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:23 smithi086 ceph-mon[57919]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:17:23.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:23 smithi083 systemd[1]: Stopping Ceph mon.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:17:23.948 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:17:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: [04/Mar/2024:16:17:23] ENGINE Bus STOPPING 2024-03-04T16:17:23.948 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:17:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: [04/Mar/2024:16:17:23] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-04T16:17:23.948 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:17:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: [04/Mar/2024:16:17:23] ENGINE Bus STOPPED 2024-03-04T16:17:23.948 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:17:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: [04/Mar/2024:16:17:23] ENGINE Bus STARTING 2024-03-04T16:17:23.948 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:17:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: CherryPy Checker: 2024-03-04T16:17:23.948 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:17:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: The Application mounted at '' has an empty config. 2024-03-04T16:17:23.948 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:17:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:17:23.949 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:17:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: [04/Mar/2024:16:17:23] ENGINE Serving on http://:::9283 2024-03-04T16:17:23.949 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:17:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: [04/Mar/2024:16:17:23] ENGINE Bus STARTED 2024-03-04T16:17:23.949 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a[64839]: 2024-03-04T16:17:23.588+0000 7fc7509cd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:17:23.949 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a[64839]: 2024-03-04T16:17:23.588+0000 7fc7509cd700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-03-04T16:17:23.949 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:23 smithi083 podman[91029]: 2024-03-04 16:17:23.69131787 +0000 UTC m=+0.232638777 container died c173a244219755ddf31bc89aef1bb32a59ef13edc120c8eeaae95d7f32389eb2 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:17:24.237 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:24 smithi083 podman[91029]: 2024-03-04 16:17:24.047520613 +0000 UTC m=+0.588841534 container cleanup c173a244219755ddf31bc89aef1bb32a59ef13edc120c8eeaae95d7f32389eb2 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-03-04T16:17:24.238 INFO:journalctl@ceph.mon.a.smithi083.stdout:Mar 04 16:17:24 smithi083 bash[91029]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-a 2024-03-04T16:17:24.526 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mon.a.service' 2024-03-04T16:17:24.577 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:17:24.577 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-04T16:17:24.578 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-04T16:17:24.578 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mon.c 2024-03-04T16:17:24.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:24 smithi083 systemd[1]: Stopping Ceph mon.c for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:17:25.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:24 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-c[67379]: 2024-03-04T16:17:24.851+0000 7ffb45d9b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:17:25.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:24 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-c[67379]: 2024-03-04T16:17:24.851+0000 7ffb45d9b700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-03-04T16:17:25.238 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:24 smithi083 podman[91248]: 2024-03-04 16:17:24.964229529 +0000 UTC m=+0.207399793 container died eff1982ba275549fdc60ee853cf593c370f1fd818123e962b23527c0ccc34723 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2) 2024-03-04T16:17:25.636 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:25 smithi083 podman[91248]: 2024-03-04 16:17:25.344250662 +0000 UTC m=+0.587420936 container cleanup eff1982ba275549fdc60ee853cf593c370f1fd818123e962b23527c0ccc34723 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-04T16:17:25.636 INFO:journalctl@ceph.mon.c.smithi083.stdout:Mar 04 16:17:25 smithi083 bash[91248]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-c 2024-03-04T16:17:25.760 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mon.c.service' 2024-03-04T16:17:25.797 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:17:25.797 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-04T16:17:25.797 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-04T16:17:25.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mon.b 2024-03-04T16:17:25.987 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:17:25 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: ::ffff:172.21.15.86 - - [04/Mar/2024:16:17:25] "GET /metrics HTTP/1.1" 200 38764 "" "Prometheus/2.33.4" 2024-03-04T16:17:26.175 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:25 smithi086 systemd[1]: Stopping Ceph mon.b for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:17:26.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:26 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-b[57915]: 2024-03-04T16:17:26.174+0000 7fddec12d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:17:26.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:26 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-b[57915]: 2024-03-04T16:17:26.174+0000 7fddec12d700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-03-04T16:17:26.484 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:26 smithi086 podman[76426]: 2024-03-04 16:17:26.307596027 +0000 UTC m=+0.308939689 container died 71557985ed5ad283b9dfab138ceafb6c19157775a91ae47f2830131208faa637 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-04T16:17:26.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:26 smithi086 podman[76426]: 2024-03-04 16:17:26.775335746 +0000 UTC m=+0.776679447 container cleanup 71557985ed5ad283b9dfab138ceafb6c19157775a91ae47f2830131208faa637 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-b, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-04T16:17:26.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 04 16:17:26 smithi086 bash[76426]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mon-b 2024-03-04T16:17:27.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mon.b.service' 2024-03-04T16:17:27.254 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:17:27.254 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-04T16:17:27.254 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-04T16:17:27.254 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mgr.y 2024-03-04T16:17:27.581 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:17:27 smithi083 systemd[1]: Stopping Ceph mgr.y for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:17:27.916 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:17:27 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y[60484]: 2024-03-04T16:17:27.581+0000 7fe8a15e7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:17:27.917 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:17:27 smithi083 podman[91369]: 2024-03-04 16:17:27.61391562 +0000 UTC m=+0.170229827 container died 164dd93609ee68c6b3736fef7f15373aa062ae504407c5f92e7f8282295a5cd2 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-04T16:17:28.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:17:27 smithi083 podman[91369]: 2024-03-04 16:17:27.999537843 +0000 UTC m=+0.555852095 container cleanup 164dd93609ee68c6b3736fef7f15373aa062ae504407c5f92e7f8282295a5cd2 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True) 2024-03-04T16:17:28.238 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 04 16:17:28 smithi083 bash[91369]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-y 2024-03-04T16:17:28.370 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mgr.y.service' 2024-03-04T16:17:28.409 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:17:28.409 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-04T16:17:28.409 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-04T16:17:28.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mgr.x 2024-03-04T16:17:28.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:17:28 smithi086 systemd[1]: Stopping Ceph mgr.x for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:17:29.106 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:17:28 smithi086 podman[76547]: 2024-03-04 16:17:28.744596213 +0000 UTC m=+0.141878012 container died 3c33100cb8a899af8230b0790d80e5c17d66903c91eb394ac0ce6be286ef587d (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:17:29.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:17:29 smithi086 podman[76547]: 2024-03-04 16:17:29.196725357 +0000 UTC m=+0.594007150 container cleanup 3c33100cb8a899af8230b0790d80e5c17d66903c91eb394ac0ce6be286ef587d (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux ) 2024-03-04T16:17:29.484 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 04 16:17:29 smithi086 bash[76547]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-mgr-x 2024-03-04T16:17:29.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@mgr.x.service' 2024-03-04T16:17:29.677 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:17:29.677 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-04T16:17:29.678 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-04T16:17:29.678 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.0 2024-03-04T16:17:29.988 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:17:29 smithi083 systemd[1]: Stopping Ceph osd.0 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:17:30.488 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:17:30 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0[70742]: 2024-03-04T16:17:30.011+0000 7f4f0d656700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:17:30.488 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:17:30 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0[70742]: 2024-03-04T16:17:30.011+0000 7f4f0d656700 -1 osd.0 153 *** Got signal Terminated *** 2024-03-04T16:17:30.488 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:17:30 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0[70742]: 2024-03-04T16:17:30.011+0000 7f4f0d656700 -1 osd.0 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:17:33.988 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:17:33 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:17:33.540Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:17:33.988 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:17:33 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:17:33.541Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-03-04T16:17:35.622 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:17:35 smithi083 podman[91487]: 2024-03-04 16:17:35.34101429 +0000 UTC m=+5.468531454 container died f7ecdc5b152057ab77f47719d440e53b494eadbb984afa1c467cbf0021db3eda (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux ) 2024-03-04T16:17:35.988 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:17:35 smithi083 podman[91487]: 2024-03-04 16:17:35.72055961 +0000 UTC m=+5.848076773 container cleanup f7ecdc5b152057ab77f47719d440e53b494eadbb984afa1c467cbf0021db3eda (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0) 2024-03-04T16:17:35.988 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:17:35 smithi083 bash[91487]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0 2024-03-04T16:17:36.328 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:17:35 smithi083 podman[91504]: 2024-03-04 16:17:35.988417286 +0000 UTC m=+0.642776192 container remove f7ecdc5b152057ab77f47719d440e53b494eadbb984afa1c467cbf0021db3eda (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-04T16:17:36.600 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:17:36 smithi083 podman[91596]: 2024-03-04 16:17:36.231517943 +0000 UTC m=+0.017268533 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:17:36.600 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:17:36 smithi083 podman[91596]: 2024-03-04 16:17:36.339129094 +0000 UTC m=+0.124879667 container create 36dd8ff269f2604fe6f0e2ab3a5a620ffed7e2c04b6134f7f8a7da33d4738d75 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-04T16:17:36.600 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:17:36 smithi083 podman[91596]: 2024-03-04 16:17:36.600172811 +0000 UTC m=+0.385923392 container init 36dd8ff269f2604fe6f0e2ab3a5a620ffed7e2c04b6134f7f8a7da33d4738d75 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222) 2024-03-04T16:17:36.891 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:17:36 smithi083 podman[91596]: 2024-03-04 16:17:36.605012731 +0000 UTC m=+0.390763307 container start 36dd8ff269f2604fe6f0e2ab3a5a620ffed7e2c04b6134f7f8a7da33d4738d75 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-04T16:17:36.891 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:17:36 smithi083 podman[91596]: 2024-03-04 16:17:36.70851216 +0000 UTC m=+0.494262741 container attach 36dd8ff269f2604fe6f0e2ab3a5a620ffed7e2c04b6134f7f8a7da33d4738d75 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-04T16:17:37.171 INFO:journalctl@ceph.osd.0.smithi083.stdout:Mar 04 16:17:36 smithi083 podman[91596]: 2024-03-04 16:17:36.891773003 +0000 UTC m=+0.677523570 container died 36dd8ff269f2604fe6f0e2ab3a5a620ffed7e2c04b6134f7f8a7da33d4738d75 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222) 2024-03-04T16:17:37.492 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.0.service' 2024-03-04T16:17:37.535 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:17:37.535 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-04T16:17:37.535 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-04T16:17:37.535 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.1 2024-03-04T16:17:37.737 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:17:37 smithi083 systemd[1]: Stopping Ceph osd.1 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:17:38.238 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:17:37 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1[73210]: 2024-03-04T16:17:37.859+0000 7f6baf9ba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:17:38.238 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:17:37 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1[73210]: 2024-03-04T16:17:37.859+0000 7f6baf9ba700 -1 osd.1 153 *** Got signal Terminated *** 2024-03-04T16:17:38.238 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:17:37 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1[73210]: 2024-03-04T16:17:37.859+0000 7f6baf9ba700 -1 osd.1 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:17:43.484 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:17:43 smithi083 podman[91702]: 2024-03-04 16:17:43.169612809 +0000 UTC m=+5.389114912 container stop 2c648d9a2b38ee92144deeb11a45d1b0558fb134988ef3c83391b000dc6874ff (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-04T16:17:43.484 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:17:43 smithi083 podman[91702]: 2024-03-04 16:17:43.195515795 +0000 UTC m=+5.415017897 container died 2c648d9a2b38ee92144deeb11a45d1b0558fb134988ef3c83391b000dc6874ff (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True) 2024-03-04T16:17:43.738 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:17:43 smithi083 podman[91702]: 2024-03-04 16:17:43.583753432 +0000 UTC m=+5.803255577 container cleanup 2c648d9a2b38ee92144deeb11a45d1b0558fb134988ef3c83391b000dc6874ff (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:17:43.738 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:17:43 smithi083 bash[91702]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1 2024-03-04T16:17:43.738 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:17:43 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=error ts=2024-03-04T16:17:43.540Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:17:43.738 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:17:43 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:17:43.541Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:17:43.738 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:17:43 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:17:43.542Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-03-04T16:17:44.237 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:17:43 smithi083 podman[91718]: 2024-03-04 16:17:43.815371867 +0000 UTC m=+0.638363447 container remove 2c648d9a2b38ee92144deeb11a45d1b0558fb134988ef3c83391b000dc6874ff (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:17:44.546 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:17:44 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:17:44.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:17:44.546 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:17:44 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:17:44.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-03-04T16:17:44.546 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:17:44 smithi083 podman[91813]: 2024-03-04 16:17:44.146012466 +0000 UTC m=+0.018362777 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:17:44.547 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:17:44 smithi083 podman[91813]: 2024-03-04 16:17:44.269447813 +0000 UTC m=+0.141798120 container create d6c741f9d4d9bf85d3256d2fdb8ef6c2c38725cafcfaebf387f20ccd7fdca792 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-04T16:17:44.547 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:17:44 smithi083 podman[91813]: 2024-03-04 16:17:44.547082999 +0000 UTC m=+0.419433318 container init d6c741f9d4d9bf85d3256d2fdb8ef6c2c38725cafcfaebf387f20ccd7fdca792 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-04T16:17:44.857 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:17:44 smithi083 podman[91813]: 2024-03-04 16:17:44.550324412 +0000 UTC m=+0.422674716 container start d6c741f9d4d9bf85d3256d2fdb8ef6c2c38725cafcfaebf387f20ccd7fdca792 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True) 2024-03-04T16:17:44.857 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:17:44 smithi083 podman[91813]: 2024-03-04 16:17:44.638724951 +0000 UTC m=+0.511075271 container attach d6c741f9d4d9bf85d3256d2fdb8ef6c2c38725cafcfaebf387f20ccd7fdca792 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, ceph=True) 2024-03-04T16:17:45.141 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 04 16:17:44 smithi083 podman[91813]: 2024-03-04 16:17:44.858582487 +0000 UTC m=+0.730932790 container died d6c741f9d4d9bf85d3256d2fdb8ef6c2c38725cafcfaebf387f20ccd7fdca792 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:17:45.406 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.1.service' 2024-03-04T16:17:45.448 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:17:45.448 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-04T16:17:45.448 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-04T16:17:45.448 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.2 2024-03-04T16:17:45.737 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:17:45 smithi083 systemd[1]: Stopping Ceph osd.2 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:17:46.238 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:17:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2[75685]: 2024-03-04T16:17:45.803+0000 7f1cd6ddb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:17:46.238 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:17:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2[75685]: 2024-03-04T16:17:45.803+0000 7f1cd6ddb700 -1 osd.2 153 *** Got signal Terminated *** 2024-03-04T16:17:46.238 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:17:45 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2[75685]: 2024-03-04T16:17:45.803+0000 7f1cd6ddb700 -1 osd.2 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:17:51.396 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:17:51 smithi083 podman[91919]: 2024-03-04 16:17:51.129307519 +0000 UTC m=+5.445076184 container died 3c971c1f3f0a4b75eeffa8719ef3649c2706cdea3aa1c0266c7812c0941f2c80 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-03-04T16:17:51.737 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:17:51 smithi083 podman[91919]: 2024-03-04 16:17:51.495213707 +0000 UTC m=+5.810982411 container cleanup 3c971c1f3f0a4b75eeffa8719ef3649c2706cdea3aa1c0266c7812c0941f2c80 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:17:51.738 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:17:51 smithi083 bash[91919]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2 2024-03-04T16:17:52.158 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:17:51 smithi083 podman[91947]: 2024-03-04 16:17:51.80492059 +0000 UTC m=+0.671056376 container remove 3c971c1f3f0a4b75eeffa8719ef3649c2706cdea3aa1c0266c7812c0941f2c80 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:17:52.411 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:17:52 smithi083 podman[92040]: 2024-03-04 16:17:52.060848708 +0000 UTC m=+0.017478512 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:17:52.411 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:17:52 smithi083 podman[92040]: 2024-03-04 16:17:52.197497669 +0000 UTC m=+0.154127532 container create d1d68b36b6bd2bdbf1355bdda5e59512ebac7c5d3b0bc0fca28b12308546029f (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:17:52.411 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:17:52 smithi083 podman[92040]: 2024-03-04 16:17:52.408445614 +0000 UTC m=+0.365075447 container init d1d68b36b6bd2bdbf1355bdda5e59512ebac7c5d3b0bc0fca28b12308546029f (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-04T16:17:52.411 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:17:52 smithi083 podman[92040]: 2024-03-04 16:17:52.411646627 +0000 UTC m=+0.368276438 container start d1d68b36b6bd2bdbf1355bdda5e59512ebac7c5d3b0bc0fca28b12308546029f (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True) 2024-03-04T16:17:52.702 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:17:52 smithi083 podman[92040]: 2024-03-04 16:17:52.516692215 +0000 UTC m=+0.473322040 container attach d1d68b36b6bd2bdbf1355bdda5e59512ebac7c5d3b0bc0fca28b12308546029f (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-04T16:17:52.988 INFO:journalctl@ceph.osd.2.smithi083.stdout:Mar 04 16:17:52 smithi083 podman[92040]: 2024-03-04 16:17:52.703561562 +0000 UTC m=+0.660191370 container died d1d68b36b6bd2bdbf1355bdda5e59512ebac7c5d3b0bc0fca28b12308546029f (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-04T16:17:53.294 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.2.service' 2024-03-04T16:17:53.337 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:17:53.337 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-04T16:17:53.337 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-04T16:17:53.337 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.3 2024-03-04T16:17:53.541 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:17:53 smithi083 systemd[1]: Stopping Ceph osd.3 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:17:53.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:17:53 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3[78174]: 2024-03-04T16:17:53.674+0000 7f1a03610700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:17:53.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:17:53 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3[78174]: 2024-03-04T16:17:53.674+0000 7f1a03610700 -1 osd.3 153 *** Got signal Terminated *** 2024-03-04T16:17:53.988 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:17:53 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3[78174]: 2024-03-04T16:17:53.674+0000 7f1a03610700 -1 osd.3 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:17:53.988 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:17:53 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=error ts=2024-03-04T16:17:53.541Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:17:53.988 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:17:53 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:17:53.542Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:17:53.989 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:17:53 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:17:53.542Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-03-04T16:17:54.737 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:17:54 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=error ts=2024-03-04T16:17:54.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-03-04T16:17:54.738 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:17:54 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:17:54.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:17:54.738 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:17:54 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:17:54.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-03-04T16:17:56.487 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:17:56 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3[78174]: 2024-03-04T16:17:56.206+0000 7f19ffc04700 -1 osd.3 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:34.412959+0000 front 2024-03-04T16:17:34.412978+0000 (oldest deadline 2024-03-04T16:17:56.113000+0000) 2024-03-04T16:17:56.733 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:17:56 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:17:56.294+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:17:56.984 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:17:56 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[61209]: 2024-03-04T16:17:56.732+0000 7f37360cd700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.572007+0000 front 2024-03-04T16:17:30.572169+0000 (oldest deadline 2024-03-04T16:17:55.772351+0000) 2024-03-04T16:17:57.488 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:17:57 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3[78174]: 2024-03-04T16:17:57.181+0000 7f19ffc04700 -1 osd.3 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:34.412959+0000 front 2024-03-04T16:17:34.412978+0000 (oldest deadline 2024-03-04T16:17:56.113000+0000) 2024-03-04T16:17:57.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:17:57 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:17:57.287+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:17:57.734 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:17:57 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:17:57.438+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.916663+0000 front 2024-03-04T16:17:30.916815+0000 (oldest deadline 2024-03-04T16:17:56.816496+0000) 2024-03-04T16:17:58.142 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:17:57 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[61209]: 2024-03-04T16:17:57.750+0000 7f37360cd700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.572007+0000 front 2024-03-04T16:17:30.572169+0000 (oldest deadline 2024-03-04T16:17:55.772351+0000) 2024-03-04T16:17:58.394 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:17:58 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:17:58.328+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:17:58.488 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:17:58 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3[78174]: 2024-03-04T16:17:58.208+0000 7f19ffc04700 -1 osd.3 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:34.412959+0000 front 2024-03-04T16:17:34.412978+0000 (oldest deadline 2024-03-04T16:17:56.113000+0000) 2024-03-04T16:17:58.733 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:17:58 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:17:58.394+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.916663+0000 front 2024-03-04T16:17:30.916815+0000 (oldest deadline 2024-03-04T16:17:56.816496+0000) 2024-03-04T16:17:59.234 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:17:58 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[61209]: 2024-03-04T16:17:58.736+0000 7f37360cd700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.572007+0000 front 2024-03-04T16:17:30.572169+0000 (oldest deadline 2024-03-04T16:17:55.772351+0000) 2024-03-04T16:17:59.324 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:17:59 smithi083 podman[92145]: 2024-03-04 16:17:59.065641138 +0000 UTC m=+5.507540441 container died 8991d467d41c30d24f0738a597f97380e7f7674de42c05dd9fe8c0a707d1599c (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-04T16:17:59.682 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:17:59 smithi083 podman[92145]: 2024-03-04 16:17:59.422745482 +0000 UTC m=+5.864644790 container cleanup 8991d467d41c30d24f0738a597f97380e7f7674de42c05dd9fe8c0a707d1599c (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-04T16:17:59.682 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:17:59 smithi083 bash[92145]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3 2024-03-04T16:17:59.691 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:17:59 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:17:59.299+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:17:59.691 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:17:59 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:17:59.426+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.916663+0000 front 2024-03-04T16:17:30.916815+0000 (oldest deadline 2024-03-04T16:17:56.816496+0000) 2024-03-04T16:17:59.984 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:17:59 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[61209]: 2024-03-04T16:17:59.729+0000 7f37360cd700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.572007+0000 front 2024-03-04T16:17:30.572169+0000 (oldest deadline 2024-03-04T16:17:55.772351+0000) 2024-03-04T16:17:59.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:17:59 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:17:59.691+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:17:59.987 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:17:59 smithi083 podman[92164]: 2024-03-04 16:17:59.682186022 +0000 UTC m=+0.614242712 container remove 8991d467d41c30d24f0738a597f97380e7f7674de42c05dd9fe8c0a707d1599c (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0) 2024-03-04T16:18:00.302 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:17:59 smithi083 podman[92256]: 2024-03-04 16:17:59.89904094 +0000 UTC m=+0.017775839 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:18:00.302 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:17:59 smithi083 podman[92256]: 2024-03-04 16:17:59.999728264 +0000 UTC m=+0.118463205 container create 199e39c4b0be9e5b945d402a744ec2f12a40c295629dbd5e585c7a0e9864a9b7 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:18:00.302 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:18:00 smithi083 podman[92256]: 2024-03-04 16:18:00.210643336 +0000 UTC m=+0.329378216 container init 199e39c4b0be9e5b945d402a744ec2f12a40c295629dbd5e585c7a0e9864a9b7 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:18:00.302 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:18:00 smithi083 podman[92256]: 2024-03-04 16:18:00.213674284 +0000 UTC m=+0.332409171 container start 199e39c4b0be9e5b945d402a744ec2f12a40c295629dbd5e585c7a0e9864a9b7 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True) 2024-03-04T16:18:00.302 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:18:00 smithi083 podman[92256]: 2024-03-04 16:18:00.302228118 +0000 UTC m=+0.420963016 container attach 199e39c4b0be9e5b945d402a744ec2f12a40c295629dbd5e585c7a0e9864a9b7 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-04T16:18:00.701 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:00 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:00.422+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.916663+0000 front 2024-03-04T16:17:30.916815+0000 (oldest deadline 2024-03-04T16:17:56.816496+0000) 2024-03-04T16:18:00.701 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:00 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:00.295+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:00.738 INFO:journalctl@ceph.osd.3.smithi083.stdout:Mar 04 16:18:00 smithi083 podman[92256]: 2024-03-04 16:18:00.506675899 +0000 UTC m=+0.625410778 container died 199e39c4b0be9e5b945d402a744ec2f12a40c295629dbd5e585c7a0e9864a9b7 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:18:00.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:00 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:00.701+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:00.984 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:00 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[61209]: 2024-03-04T16:18:00.748+0000 7f37360cd700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.572007+0000 front 2024-03-04T16:17:30.572169+0000 (oldest deadline 2024-03-04T16:17:55.772351+0000) 2024-03-04T16:18:01.076 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.3.service' 2024-03-04T16:18:01.118 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:18:01.118 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-04T16:18:01.119 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-04T16:18:01.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.4 2024-03-04T16:18:01.437 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:01 smithi086 systemd[1]: Stopping Ceph osd.4 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:18:01.437 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:01 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:01.310+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:01.715 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:01 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:01.437+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.916663+0000 front 2024-03-04T16:17:30.916815+0000 (oldest deadline 2024-03-04T16:17:56.816496+0000) 2024-03-04T16:18:01.716 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:01 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[61209]: 2024-03-04T16:18:01.589+0000 7f373dae1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:18:01.716 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:01 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[61209]: 2024-03-04T16:18:01.589+0000 7f373dae1700 -1 osd.4 153 *** Got signal Terminated *** 2024-03-04T16:18:01.716 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:01 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[61209]: 2024-03-04T16:18:01.589+0000 7f373dae1700 -1 osd.4 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:18:01.716 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:01 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[61209]: 2024-03-04T16:18:01.716+0000 7f37360cd700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.572007+0000 front 2024-03-04T16:17:30.572169+0000 (oldest deadline 2024-03-04T16:17:55.772351+0000) 2024-03-04T16:18:01.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:01 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:01.716+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:02.696 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:02 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:02.329+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:02.696 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:02 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:02.483+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.916663+0000 front 2024-03-04T16:17:30.916815+0000 (oldest deadline 2024-03-04T16:17:56.816496+0000) 2024-03-04T16:18:02.984 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:02 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[61209]: 2024-03-04T16:18:02.744+0000 7f37360cd700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.572007+0000 front 2024-03-04T16:17:30.572169+0000 (oldest deadline 2024-03-04T16:17:55.772351+0000) 2024-03-04T16:18:02.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:02 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:02.696+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:03.729 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:03 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:03.284+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:03.730 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:03 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:03.466+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.916663+0000 front 2024-03-04T16:17:30.916815+0000 (oldest deadline 2024-03-04T16:17:56.816496+0000) 2024-03-04T16:18:03.737 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:03 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=error ts=2024-03-04T16:18:03.542Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:18:03.738 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:03 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:18:03.543Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:18:03.738 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:03 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:18:03.544Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-03-04T16:18:03.984 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:03 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[61209]: 2024-03-04T16:18:03.747+0000 7f37360cd700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.572007+0000 front 2024-03-04T16:17:30.572169+0000 (oldest deadline 2024-03-04T16:17:55.772351+0000) 2024-03-04T16:18:03.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:03 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:03.730+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:03.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:03 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:03.730+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:04.700 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:04 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:04.505+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.916663+0000 front 2024-03-04T16:17:30.916815+0000 (oldest deadline 2024-03-04T16:17:56.816496+0000) 2024-03-04T16:18:04.703 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:04 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:04.281+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:04.737 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:04 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=error ts=2024-03-04T16:18:04.242Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:18:04.738 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:04 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:18:04.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:18:04.738 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:04 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:18:04.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-03-04T16:18:04.984 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:04 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[61209]: 2024-03-04T16:18:04.700+0000 7f37360cd700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.572007+0000 front 2024-03-04T16:17:30.572169+0000 (oldest deadline 2024-03-04T16:17:55.772351+0000) 2024-03-04T16:18:04.984 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:04 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[61209]: 2024-03-04T16:18:04.700+0000 7f37360cd700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:41.673294+0000 front 2024-03-04T16:17:41.673281+0000 (oldest deadline 2024-03-04T16:18:04.573256+0000) 2024-03-04T16:18:04.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:04 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:04.748+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:04.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:04 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:04.748+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:05.543 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:05 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:05.287+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:05.544 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:05 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:05.287+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:05.984 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:05 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:05.544+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.916663+0000 front 2024-03-04T16:17:30.916815+0000 (oldest deadline 2024-03-04T16:17:56.816496+0000) 2024-03-04T16:18:05.984 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:05 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[61209]: 2024-03-04T16:18:05.703+0000 7f37360cd700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.572007+0000 front 2024-03-04T16:17:30.572169+0000 (oldest deadline 2024-03-04T16:17:55.772351+0000) 2024-03-04T16:18:05.984 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:05 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4[61209]: 2024-03-04T16:18:05.703+0000 7f37360cd700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:41.673294+0000 front 2024-03-04T16:17:41.673281+0000 (oldest deadline 2024-03-04T16:18:04.573256+0000) 2024-03-04T16:18:05.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:05 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:05.724+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:05.985 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:05 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:05.724+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:06.734 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:06 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:06.571+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.916663+0000 front 2024-03-04T16:17:30.916815+0000 (oldest deadline 2024-03-04T16:17:56.816496+0000) 2024-03-04T16:18:06.734 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:06 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:06.571+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.317701+0000 front 2024-03-04T16:17:40.317661+0000 (oldest deadline 2024-03-04T16:18:06.217634+0000) 2024-03-04T16:18:06.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:06 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:06.333+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:06.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:06 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:06.333+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:07.088 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:06 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:06.755+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:07.088 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:06 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:06.755+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:07.473 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:07 smithi086 podman[76694]: 2024-03-04 16:18:07.088687924 +0000 UTC m=+5.773823603 container died 912383028c724e2322ef8f858df94d326c1ce58da8b0d812c62f000d30e6cf81 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-04T16:18:07.473 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:07 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:07.323+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:07.474 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:07 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:07.323+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:07.734 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:07 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:07.590+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.916663+0000 front 2024-03-04T16:17:30.916815+0000 (oldest deadline 2024-03-04T16:17:56.816496+0000) 2024-03-04T16:18:07.734 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:07 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:07.590+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.317701+0000 front 2024-03-04T16:17:40.317661+0000 (oldest deadline 2024-03-04T16:18:06.217634+0000) 2024-03-04T16:18:07.734 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:07 smithi086 podman[76694]: 2024-03-04 16:18:07.473872435 +0000 UTC m=+6.159008115 container cleanup 912383028c724e2322ef8f858df94d326c1ce58da8b0d812c62f000d30e6cf81 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-04T16:18:07.734 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:07 smithi086 bash[76694]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4 2024-03-04T16:18:07.734 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:07 smithi086 podman[76711]: 2024-03-04 16:18:07.717314537 +0000 UTC m=+0.626107308 container remove 912383028c724e2322ef8f858df94d326c1ce58da8b0d812c62f000d30e6cf81 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-04T16:18:08.041 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:07 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:07.770+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:08.041 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:07 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:07.770+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:08.320 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:08 smithi086 podman[76805]: 2024-03-04 16:18:07.943793753 +0000 UTC m=+0.018457526 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:18:08.320 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:08 smithi086 podman[76805]: 2024-03-04 16:18:08.043109648 +0000 UTC m=+0.117773461 container create 98fdb12504a0c36fdf702d1b72e7c6520fd8a443fb2509ee622cbf05567e9dd3 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, ceph=True) 2024-03-04T16:18:08.320 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:08 smithi086 podman[76805]: 2024-03-04 16:18:08.245335575 +0000 UTC m=+0.319999344 container init 98fdb12504a0c36fdf702d1b72e7c6520fd8a443fb2509ee622cbf05567e9dd3 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-03-04T16:18:08.320 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:08 smithi086 podman[76805]: 2024-03-04 16:18:08.248388144 +0000 UTC m=+0.323051913 container start 98fdb12504a0c36fdf702d1b72e7c6520fd8a443fb2509ee622cbf05567e9dd3 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:18:08.320 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:08 smithi086 podman[76805]: 2024-03-04 16:18:08.320368539 +0000 UTC m=+0.395032308 container attach 98fdb12504a0c36fdf702d1b72e7c6520fd8a443fb2509ee622cbf05567e9dd3 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD) 2024-03-04T16:18:08.597 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:08 smithi086 podman[76805]: 2024-03-04 16:18:08.538921694 +0000 UTC m=+0.613585463 container died 98fdb12504a0c36fdf702d1b72e7c6520fd8a443fb2509ee622cbf05567e9dd3 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-04T16:18:08.597 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:08 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:08.369+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:08.597 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:08 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:08.369+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:08.937 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:08 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:08.726+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:08.937 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:08 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:08.726+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:08.937 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:08 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:08.596+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.916663+0000 front 2024-03-04T16:17:30.916815+0000 (oldest deadline 2024-03-04T16:17:56.816496+0000) 2024-03-04T16:18:08.937 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:08 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:08.596+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.317701+0000 front 2024-03-04T16:17:40.317661+0000 (oldest deadline 2024-03-04T16:18:06.217634+0000) 2024-03-04T16:18:09.174 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.4.service' 2024-03-04T16:18:09.200 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:09 smithi086 podman[76831]: 2024-03-04 16:18:09.149541062 +0000 UTC m=+0.604598760 container remove 98fdb12504a0c36fdf702d1b72e7c6520fd8a443fb2509ee622cbf05567e9dd3 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-04T16:18:09.200 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:09 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-04T16:18:09.200 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:09 smithi086 systemd[1]: Stopped Ceph osd.4 for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:18:09.200 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 04 16:18:09 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.4.service: Consumed 6.249s CPU time. 2024-03-04T16:18:09.216 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:18:09.216 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-04T16:18:09.216 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-04T16:18:09.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.5 2024-03-04T16:18:09.484 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:09 smithi086 systemd[1]: Stopping Ceph osd.5 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:18:09.484 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:09 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:09.405+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:09.484 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:09 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:09.405+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:09.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:09 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:09.727+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:09.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:09 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:09.727+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:09.984 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:09 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:09.571+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.916663+0000 front 2024-03-04T16:17:30.916815+0000 (oldest deadline 2024-03-04T16:17:56.816496+0000) 2024-03-04T16:18:09.984 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:09 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:09.571+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.317701+0000 front 2024-03-04T16:17:40.317661+0000 (oldest deadline 2024-03-04T16:18:06.217634+0000) 2024-03-04T16:18:09.984 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:09 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:09.595+0000 7f71c88a5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:18:09.984 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:09 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:09.595+0000 7f71c88a5700 -1 osd.5 153 *** Got signal Terminated *** 2024-03-04T16:18:09.984 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:09 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:09.595+0000 7f71c88a5700 -1 osd.5 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:18:10.731 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:10 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:10.540+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.916663+0000 front 2024-03-04T16:17:30.916815+0000 (oldest deadline 2024-03-04T16:17:56.816496+0000) 2024-03-04T16:18:10.732 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:10 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:10.540+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.317701+0000 front 2024-03-04T16:17:40.317661+0000 (oldest deadline 2024-03-04T16:18:06.217634+0000) 2024-03-04T16:18:10.732 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:10 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:10.451+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:10.732 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:10 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:10.451+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:10.983 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:10 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:10.730+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:10.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:10 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:10.730+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:11.694 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:11.491+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.916663+0000 front 2024-03-04T16:17:30.916815+0000 (oldest deadline 2024-03-04T16:17:56.816496+0000) 2024-03-04T16:18:11.694 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:11.491+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.317701+0000 front 2024-03-04T16:17:40.317661+0000 (oldest deadline 2024-03-04T16:18:06.217634+0000) 2024-03-04T16:18:11.695 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:11.403+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:11.695 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:11.403+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:11.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:11.693+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:11.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:11 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:11.693+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:12.714 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:12 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:12.502+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.916663+0000 front 2024-03-04T16:17:30.916815+0000 (oldest deadline 2024-03-04T16:17:56.816496+0000) 2024-03-04T16:18:12.714 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:12 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:12.502+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.317701+0000 front 2024-03-04T16:17:40.317661+0000 (oldest deadline 2024-03-04T16:18:06.217634+0000) 2024-03-04T16:18:12.714 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:12 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:12.381+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:12.715 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:12 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:12.381+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:12.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:12 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:12.713+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:12.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:12 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:12.713+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:12.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:12 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:12.713+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.491293+0000 front 2024-03-04T16:17:49.491278+0000 (oldest deadline 2024-03-04T16:18:11.791052+0000) 2024-03-04T16:18:13.693 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:13 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:13.533+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.916663+0000 front 2024-03-04T16:17:30.916815+0000 (oldest deadline 2024-03-04T16:17:56.816496+0000) 2024-03-04T16:18:13.693 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:13 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:13.533+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.317701+0000 front 2024-03-04T16:17:40.317661+0000 (oldest deadline 2024-03-04T16:18:06.217634+0000) 2024-03-04T16:18:13.693 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:13 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:13.420+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:13.693 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:13 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:13.420+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:13.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:13 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:13.692+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:13.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:13 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:13.692+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:13.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:13 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:13.692+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.491293+0000 front 2024-03-04T16:17:49.491278+0000 (oldest deadline 2024-03-04T16:18:11.791052+0000) 2024-03-04T16:18:13.987 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:13 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=error ts=2024-03-04T16:18:13.543Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:18:13.987 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:13 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:18:13.543Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:18:13.988 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:13 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:18:13.544Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-03-04T16:18:14.488 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:14 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=error ts=2024-03-04T16:18:14.242Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:18:14.488 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:14 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:18:14.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:18:14.488 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:14 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:18:14.243Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-03-04T16:18:14.734 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:14.563+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.916663+0000 front 2024-03-04T16:17:30.916815+0000 (oldest deadline 2024-03-04T16:17:56.816496+0000) 2024-03-04T16:18:14.734 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:14.563+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.317701+0000 front 2024-03-04T16:17:40.317661+0000 (oldest deadline 2024-03-04T16:18:06.217634+0000) 2024-03-04T16:18:14.734 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5[63532]: 2024-03-04T16:18:14.563+0000 7f71c0e91700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:48.919358+0000 front 2024-03-04T16:17:48.919393+0000 (oldest deadline 2024-03-04T16:18:13.619350+0000) 2024-03-04T16:18:14.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:14.696+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:14.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:14.696+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:14.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:14.696+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.491293+0000 front 2024-03-04T16:17:49.491278+0000 (oldest deadline 2024-03-04T16:18:11.791052+0000) 2024-03-04T16:18:14.735 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:14.446+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:14.735 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:14.446+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:14.735 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:14.446+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.041102+0000 front 2024-03-04T16:17:49.041137+0000 (oldest deadline 2024-03-04T16:18:14.341065+0000) 2024-03-04T16:18:15.346 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:15 smithi086 podman[76910]: 2024-03-04 16:18:15.084917691 +0000 UTC m=+5.624952004 container died 53215200f312ac0161d71dc5189f8f7ffdc06976d00fda29047f2ddd3268a093 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD) 2024-03-04T16:18:15.676 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:15 smithi086 podman[76910]: 2024-03-04 16:18:15.545028429 +0000 UTC m=+6.085062787 container cleanup 53215200f312ac0161d71dc5189f8f7ffdc06976d00fda29047f2ddd3268a093 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:18:15.676 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:15 smithi086 bash[76910]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5 2024-03-04T16:18:15.676 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:15 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:15.494+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:15.676 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:15 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:15.494+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:15.676 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:15 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:15.494+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.041102+0000 front 2024-03-04T16:17:49.041137+0000 (oldest deadline 2024-03-04T16:18:14.341065+0000) 2024-03-04T16:18:15.984 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:15 smithi086 podman[76938]: 2024-03-04 16:18:15.821101805 +0000 UTC m=+0.729048915 container remove 53215200f312ac0161d71dc5189f8f7ffdc06976d00fda29047f2ddd3268a093 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, org.label-schema.build-date=20240222) 2024-03-04T16:18:15.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:15 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:15.675+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:15.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:15 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:15.675+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:15.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:15 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:15.675+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.491293+0000 front 2024-03-04T16:17:49.491278+0000 (oldest deadline 2024-03-04T16:18:11.791052+0000) 2024-03-04T16:18:16.407 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:16 smithi086 podman[77030]: 2024-03-04 16:18:16.113577311 +0000 UTC m=+0.104475050 container create 07b258383cdd8681161b507fe55c2c9c7e92f5f62ae384371e5112bdfd578c5a (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True) 2024-03-04T16:18:16.407 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:16 smithi086 podman[77030]: 2024-03-04 16:18:16.026903919 +0000 UTC m=+0.017801616 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:18:16.407 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:16 smithi086 podman[77030]: 2024-03-04 16:18:16.332625442 +0000 UTC m=+0.323523160 container init 07b258383cdd8681161b507fe55c2c9c7e92f5f62ae384371e5112bdfd578c5a (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:18:16.408 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:16 smithi086 podman[77030]: 2024-03-04 16:18:16.335876336 +0000 UTC m=+0.326774033 container start 07b258383cdd8681161b507fe55c2c9c7e92f5f62ae384371e5112bdfd578c5a (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-04T16:18:16.408 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:16 smithi086 podman[77030]: 2024-03-04 16:18:16.407287057 +0000 UTC m=+0.398184768 container attach 07b258383cdd8681161b507fe55c2c9c7e92f5f62ae384371e5112bdfd578c5a (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-04T16:18:16.704 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 16:18:16 smithi086 podman[77030]: 2024-03-04 16:18:16.62784008 +0000 UTC m=+0.618737777 container died 07b258383cdd8681161b507fe55c2c9c7e92f5f62ae384371e5112bdfd578c5a (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, org.label-schema.vendor=CentOS) 2024-03-04T16:18:16.705 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:16 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:16.490+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:16.705 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:16 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:16.490+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:16.705 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:16 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:16.490+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.041102+0000 front 2024-03-04T16:17:49.041137+0000 (oldest deadline 2024-03-04T16:18:14.341065+0000) 2024-03-04T16:18:16.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:16 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:16.703+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:16.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:16 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:16.703+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:16.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:16 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:16.703+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.491293+0000 front 2024-03-04T16:17:49.491278+0000 (oldest deadline 2024-03-04T16:18:11.791052+0000) 2024-03-04T16:18:17.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.5.service' 2024-03-04T16:18:17.305 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:18:17.306 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-04T16:18:17.306 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-04T16:18:17.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.6 2024-03-04T16:18:17.664 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:17 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:17.460+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:17.664 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:17 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:17.461+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:17.664 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:17 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:17.461+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.041102+0000 front 2024-03-04T16:17:49.041137+0000 (oldest deadline 2024-03-04T16:18:14.341065+0000) 2024-03-04T16:18:17.664 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:17 smithi086 systemd[1]: Stopping Ceph osd.6 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:18:17.664 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:17 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:17.663+0000 7f0e6cabd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:18:17.664 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:17 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:17.663+0000 7f0e6cabd700 -1 osd.6 153 *** Got signal Terminated *** 2024-03-04T16:18:17.664 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:17 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:17.663+0000 7f0e6cabd700 -1 osd.6 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:18:17.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:17 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:17.746+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:17.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:17 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:17.746+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:17.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:17 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:17.746+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.491293+0000 front 2024-03-04T16:17:49.491278+0000 (oldest deadline 2024-03-04T16:18:11.791052+0000) 2024-03-04T16:18:18.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:18.723+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:18.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:18.723+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:18.984 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:18.723+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.491293+0000 front 2024-03-04T16:17:49.491278+0000 (oldest deadline 2024-03-04T16:18:11.791052+0000) 2024-03-04T16:18:18.984 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:18.500+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:18.984 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:18.500+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:18.985 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:18 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:18.500+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.041102+0000 front 2024-03-04T16:17:49.041137+0000 (oldest deadline 2024-03-04T16:18:14.341065+0000) 2024-03-04T16:18:19.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:19 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:19.675+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:19.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:19 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:19.675+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:19.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:19 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:19.675+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.491293+0000 front 2024-03-04T16:17:49.491278+0000 (oldest deadline 2024-03-04T16:18:11.791052+0000) 2024-03-04T16:18:19.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:19 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:19.476+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:19.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:19 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:19.476+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:19.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:19 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:19.476+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.041102+0000 front 2024-03-04T16:17:49.041137+0000 (oldest deadline 2024-03-04T16:18:14.341065+0000) 2024-03-04T16:18:20.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:20 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:20.461+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:20.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:20 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:20.461+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:20.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:20 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:20.462+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.041102+0000 front 2024-03-04T16:17:49.041137+0000 (oldest deadline 2024-03-04T16:18:14.341065+0000) 2024-03-04T16:18:20.735 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:20 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:20.653+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:20.735 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:20 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:20.653+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:20.735 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:20 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:20.653+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.491293+0000 front 2024-03-04T16:17:49.491278+0000 (oldest deadline 2024-03-04T16:18:11.791052+0000) 2024-03-04T16:18:20.735 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:20 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:20.653+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-03-04T16:17:55.292084+0000 front 2024-03-04T16:17:55.292010+0000 (oldest deadline 2024-03-04T16:18:20.591984+0000) 2024-03-04T16:18:21.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:21.458+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:21.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:21.458+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:21.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:21.458+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.041102+0000 front 2024-03-04T16:17:49.041137+0000 (oldest deadline 2024-03-04T16:18:14.341065+0000) 2024-03-04T16:18:21.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:21.684+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:21.735 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:21.684+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:21.735 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:21.684+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.491293+0000 front 2024-03-04T16:17:49.491278+0000 (oldest deadline 2024-03-04T16:18:11.791052+0000) 2024-03-04T16:18:21.735 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:21 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:21.684+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-03-04T16:17:55.292084+0000 front 2024-03-04T16:17:55.292010+0000 (oldest deadline 2024-03-04T16:18:20.591984+0000) 2024-03-04T16:18:22.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:22 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:22.477+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:22.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:22 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:22.477+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:22.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:22 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:22.477+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.041102+0000 front 2024-03-04T16:17:49.041137+0000 (oldest deadline 2024-03-04T16:18:14.341065+0000) 2024-03-04T16:18:22.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:22 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:22.648+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:33.388209+0000 front 2024-03-04T16:17:33.388290+0000 (oldest deadline 2024-03-04T16:17:59.288088+0000) 2024-03-04T16:18:22.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:22 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:22.648+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:42.789002+0000 front 2024-03-04T16:17:42.789014+0000 (oldest deadline 2024-03-04T16:18:03.288885+0000) 2024-03-04T16:18:22.735 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:22 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:22.648+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.491293+0000 front 2024-03-04T16:17:49.491278+0000 (oldest deadline 2024-03-04T16:18:11.791052+0000) 2024-03-04T16:18:22.735 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:22 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6[65896]: 2024-03-04T16:18:22.648+0000 7f0e690b1700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-03-04T16:17:55.292084+0000 front 2024-03-04T16:17:55.292010+0000 (oldest deadline 2024-03-04T16:18:20.591984+0000) 2024-03-04T16:18:23.396 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:23 smithi086 podman[77137]: 2024-03-04 16:18:23.0072339 +0000 UTC m=+5.475103152 container died a4584e5b304436cf1961578cb306acc172287f9756df7e374440cac43300d828 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:18:23.396 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:23 smithi086 podman[77137]: 2024-03-04 16:18:23.396600832 +0000 UTC m=+5.864470106 container cleanup a4584e5b304436cf1961578cb306acc172287f9756df7e374440cac43300d828 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:18:23.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:23 smithi086 bash[77137]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6 2024-03-04T16:18:23.734 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:23 smithi086 podman[77154]: 2024-03-04 16:18:23.606160061 +0000 UTC m=+0.597216098 container remove a4584e5b304436cf1961578cb306acc172287f9756df7e374440cac43300d828 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:18:23.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:23 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:23.499+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:23.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:23 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:23.499+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:23.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:23 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:23.499+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.041102+0000 front 2024-03-04T16:17:49.041137+0000 (oldest deadline 2024-03-04T16:18:14.341065+0000) 2024-03-04T16:18:23.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:23 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:23.499+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-03-04T16:17:54.842124+0000 front 2024-03-04T16:17:54.842102+0000 (oldest deadline 2024-03-04T16:18:23.042374+0000) 2024-03-04T16:18:23.988 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=error ts=2024-03-04T16:18:23.544Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-03-04T16:18:23.988 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:18:23.544Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:18:23.988 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:23 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:18:23.545Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-03-04T16:18:24.210 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:23 smithi086 podman[77251]: 2024-03-04 16:18:23.932282619 +0000 UTC m=+0.109879579 container create 4e7c04828d1d0aedf686d36608d5c7d3e3e874617320973b978ba4757e78b784 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-04T16:18:24.210 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:23 smithi086 podman[77251]: 2024-03-04 16:18:23.840926764 +0000 UTC m=+0.018523741 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:18:24.210 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:24 smithi086 podman[77251]: 2024-03-04 16:18:24.134927509 +0000 UTC m=+0.312524479 container init 4e7c04828d1d0aedf686d36608d5c7d3e3e874617320973b978ba4757e78b784 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True) 2024-03-04T16:18:24.210 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:24 smithi086 podman[77251]: 2024-03-04 16:18:24.137925005 +0000 UTC m=+0.315521965 container start 4e7c04828d1d0aedf686d36608d5c7d3e3e874617320973b978ba4757e78b784 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-04T16:18:24.484 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:24 smithi086 podman[77251]: 2024-03-04 16:18:24.209696451 +0000 UTC m=+0.387293412 container attach 4e7c04828d1d0aedf686d36608d5c7d3e3e874617320973b978ba4757e78b784 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-04T16:18:24.484 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 04 16:18:24 smithi086 podman[77251]: 2024-03-04 16:18:24.438711118 +0000 UTC m=+0.616308079 container died 4e7c04828d1d0aedf686d36608d5c7d3e3e874617320973b978ba4757e78b784 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222) 2024-03-04T16:18:24.738 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:24 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=error ts=2024-03-04T16:18:24.243Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-03-04T16:18:24.738 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:24 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:18:24.243Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:18:24.738 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:24 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:18:24.244Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-03-04T16:18:24.984 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:24 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:24.515+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:24.984 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:24 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:24.515+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:24.984 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:24 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:24.515+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.041102+0000 front 2024-03-04T16:17:49.041137+0000 (oldest deadline 2024-03-04T16:18:14.341065+0000) 2024-03-04T16:18:24.984 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:24 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:24.515+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-03-04T16:17:54.842124+0000 front 2024-03-04T16:17:54.842102+0000 (oldest deadline 2024-03-04T16:18:23.042374+0000) 2024-03-04T16:18:25.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.6.service' 2024-03-04T16:18:25.155 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:18:25.155 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-04T16:18:25.155 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-04T16:18:25.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.7 2024-03-04T16:18:25.473 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:25 smithi086 systemd[1]: Stopping Ceph osd.7 for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:18:25.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:25 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:25.473+0000 7fa5a5fe8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:18:25.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:25 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:25.473+0000 7fa5a5fe8700 -1 osd.7 153 *** Got signal Terminated *** 2024-03-04T16:18:25.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:25 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:25.473+0000 7fa5a5fe8700 -1 osd.7 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:18:25.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:25 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:25.550+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:25.735 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:25 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:25.550+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:25.735 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:25 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:25.550+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.041102+0000 front 2024-03-04T16:17:49.041137+0000 (oldest deadline 2024-03-04T16:18:14.341065+0000) 2024-03-04T16:18:25.735 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:25 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:25.550+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-03-04T16:17:54.842124+0000 front 2024-03-04T16:17:54.842102+0000 (oldest deadline 2024-03-04T16:18:23.042374+0000) 2024-03-04T16:18:26.984 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:26 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:26.505+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:26.984 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:26 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:26.505+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:26.984 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:26 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:26.505+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.041102+0000 front 2024-03-04T16:17:49.041137+0000 (oldest deadline 2024-03-04T16:18:14.341065+0000) 2024-03-04T16:18:26.984 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:26 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:26.505+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-03-04T16:17:54.842124+0000 front 2024-03-04T16:17:54.842102+0000 (oldest deadline 2024-03-04T16:18:23.042374+0000) 2024-03-04T16:18:27.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:27.507+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:27.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:27.507+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:27.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:27.507+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.041102+0000 front 2024-03-04T16:17:49.041137+0000 (oldest deadline 2024-03-04T16:18:14.341065+0000) 2024-03-04T16:18:27.734 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:27.507+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-03-04T16:17:54.842124+0000 front 2024-03-04T16:17:54.842102+0000 (oldest deadline 2024-03-04T16:18:23.042374+0000) 2024-03-04T16:18:27.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:18:27.481Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.83:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.83:8765: connect: connection refused" 2024-03-04T16:18:27.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:18:27.482Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.83:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.83:8765: connect: connection refused" 2024-03-04T16:18:27.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:18:27.484Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.83:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.83:8765: connect: connection refused" 2024-03-04T16:18:27.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:18:27.485Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.83:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.83:8765: connect: connection refused" 2024-03-04T16:18:27.735 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:27 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:18:27.485Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.83:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.83:8765: connect: connection refused" 2024-03-04T16:18:28.978 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:28 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:28.538+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:28.979 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:28 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:28.538+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:28.979 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:28 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:28.538+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.041102+0000 front 2024-03-04T16:17:49.041137+0000 (oldest deadline 2024-03-04T16:18:14.341065+0000) 2024-03-04T16:18:28.979 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:28 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:28.538+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-03-04T16:17:54.842124+0000 front 2024-03-04T16:17:54.842102+0000 (oldest deadline 2024-03-04T16:18:23.042374+0000) 2024-03-04T16:18:28.979 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:28 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:28.538+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.86:6804 osd.4 since back 2024-03-04T16:18:03.543282+0000 front 2024-03-04T16:18:03.543334+0000 (oldest deadline 2024-03-04T16:18:28.243340+0000) 2024-03-04T16:18:29.878 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:29.525+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-03-04T16:17:30.339077+0000 front 2024-03-04T16:17:30.339151+0000 (oldest deadline 2024-03-04T16:17:55.638899+0000) 2024-03-04T16:18:29.878 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:29.525+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-03-04T16:17:40.340013+0000 front 2024-03-04T16:17:40.339976+0000 (oldest deadline 2024-03-04T16:18:05.039799+0000) 2024-03-04T16:18:29.878 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:29.525+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-03-04T16:17:49.041102+0000 front 2024-03-04T16:17:49.041137+0000 (oldest deadline 2024-03-04T16:18:14.341065+0000) 2024-03-04T16:18:29.879 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:29.525+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-03-04T16:17:54.842124+0000 front 2024-03-04T16:17:54.842102+0000 (oldest deadline 2024-03-04T16:18:23.042374+0000) 2024-03-04T16:18:29.879 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:29 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7[68233]: 2024-03-04T16:18:29.525+0000 7fa59e5d4700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.86:6804 osd.4 since back 2024-03-04T16:18:03.543282+0000 front 2024-03-04T16:18:03.543334+0000 (oldest deadline 2024-03-04T16:18:28.243340+0000) 2024-03-04T16:18:31.199 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:30 smithi086 podman[77355]: 2024-03-04 16:18:30.929803189 +0000 UTC m=+5.559863298 container died 881253b7c5f18f8f64125d5abac22cb37849a96140b4ccb176fd97b379a4980d (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-03-04T16:18:31.484 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:31 smithi086 podman[77355]: 2024-03-04 16:18:31.290562305 +0000 UTC m=+5.920622421 container cleanup 881253b7c5f18f8f64125d5abac22cb37849a96140b4ccb176fd97b379a4980d (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-04T16:18:31.484 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:31 smithi086 bash[77355]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7 2024-03-04T16:18:31.900 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:31 smithi086 podman[77384]: 2024-03-04 16:18:31.575037815 +0000 UTC m=+0.636707158 container remove 881253b7c5f18f8f64125d5abac22cb37849a96140b4ccb176fd97b379a4980d (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-04T16:18:32.169 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:31 smithi086 podman[77476]: 2024-03-04 16:18:31.900788581 +0000 UTC m=+0.113441002 container create 743ba1f331ab017662530ce5b14890c6450da9ed00de1f08b38a4780b9e6c867 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222) 2024-03-04T16:18:32.169 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:31 smithi086 podman[77476]: 2024-03-04 16:18:31.805201364 +0000 UTC m=+0.017853739 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T16:18:32.169 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:32 smithi086 podman[77476]: 2024-03-04 16:18:32.095151037 +0000 UTC m=+0.307803440 container init 743ba1f331ab017662530ce5b14890c6450da9ed00de1f08b38a4780b9e6c867 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux ) 2024-03-04T16:18:32.169 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:32 smithi086 podman[77476]: 2024-03-04 16:18:32.098400273 +0000 UTC m=+0.311052650 container start 743ba1f331ab017662530ce5b14890c6450da9ed00de1f08b38a4780b9e6c867 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240222, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-04T16:18:32.170 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:32 smithi086 podman[77476]: 2024-03-04 16:18:32.169498713 +0000 UTC m=+0.382151101 container attach 743ba1f331ab017662530ce5b14890c6450da9ed00de1f08b38a4780b9e6c867 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-04T16:18:32.484 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:32 smithi086 podman[77476]: 2024-03-04 16:18:32.386778566 +0000 UTC m=+0.599430946 container died 743ba1f331ab017662530ce5b14890c6450da9ed00de1f08b38a4780b9e6c867 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-04T16:18:32.939 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 04 16:18:32 smithi086 podman[77502]: 2024-03-04 16:18:32.922543184 +0000 UTC m=+0.530123217 container remove 743ba1f331ab017662530ce5b14890c6450da9ed00de1f08b38a4780b9e6c867 (image=quay.ceph.io/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-04T16:18:32.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@osd.7.service' 2024-03-04T16:18:32.987 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:18:32.987 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-04T16:18:32.988 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-04T16:18:32.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@prometheus.a 2024-03-04T16:18:33.194 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:33 smithi086 systemd[1]: Stopping Ceph prometheus.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:18:33.194 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:18:33.184Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-04T16:18:33.194 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:18:33.184Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-04T16:18:33.195 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:18:33.184Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-04T16:18:33.195 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:18:33.184Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-04T16:18:33.195 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:18:33.184Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-04T16:18:33.195 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:18:33.184Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-04T16:18:33.195 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:18:33.184Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-04T16:18:33.195 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:18:33.184Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-04T16:18:33.195 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:18:33.184Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-04T16:18:33.195 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:18:33.184Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-04T16:18:33.195 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:18:33.184Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-04T16:18:33.195 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:33 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a[53817]: ts=2024-03-04T16:18:33.185Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-04T16:18:33.195 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:33 smithi086 podman[77554]: 2024-03-04 16:18:33.194528695 +0000 UTC m=+0.118567190 container died b563a246eeac8f2acade5c23b28c5f50df43605df1eeb125e485ac1cd20e25e6 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:18:33.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:33 smithi086 podman[77554]: 2024-03-04 16:18:33.558734006 +0000 UTC m=+0.482772504 container cleanup b563a246eeac8f2acade5c23b28c5f50df43605df1eeb125e485ac1cd20e25e6 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:18:33.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:33 smithi086 bash[77554]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus-a 2024-03-04T16:18:33.734 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 04 16:18:33 smithi086 bash[77582]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-prometheus.a" found: no such container 2024-03-04T16:18:33.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@prometheus.a.service' 2024-03-04T16:18:33.871 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:18:33.871 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-04T16:18:33.871 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-04T16:18:33.903 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:33 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=error ts=2024-03-04T16:18:33.544Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:18:33.903 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:33 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:18:33.545Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:18:33.903 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:33 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:18:33.545Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-03-04T16:18:34.524 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:34 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=error ts=2024-03-04T16:18:34.244Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-03-04T16:18:34.524 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:34 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:18:34.244Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-03-04T16:18:34.525 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:34 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=warn ts=2024-03-04T16:18:34.244Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-03-04T16:18:35.488 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:18:35 smithi083 systemd[1]: Stopping Ceph node-exporter.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:18:35.774 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:18:35 smithi083 podman[92624]: 2024-03-04 16:18:35.508848887 +0000 UTC m=+0.111198681 container died 9bf54b46064062273c6ec8a493b9c8ba473508399d146c29305015aca7f748d5 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T16:18:36.132 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:18:35 smithi083 podman[92624]: 2024-03-04 16:18:35.873490739 +0000 UTC m=+0.475840581 container cleanup 9bf54b46064062273c6ec8a493b9c8ba473508399d146c29305015aca7f748d5 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T16:18:36.132 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:18:35 smithi083 bash[92624]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a 2024-03-04T16:18:36.132 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:18:35 smithi083 bash[92649]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter.a" found: no such container 2024-03-04T16:18:36.435 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:36 smithi083 systemd[1]: Stopping Ceph alertmanager.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:18:36.435 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:18:36 smithi083 podman[92638]: 2024-03-04 16:18:36.132558702 +0000 UTC m=+0.625758289 container remove 9bf54b46064062273c6ec8a493b9c8ba473508399d146c29305015aca7f748d5 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T16:18:36.435 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:18:36 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-04T16:18:36.435 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:18:36 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-04T16:18:36.435 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:18:36 smithi083 systemd[1]: Stopped Ceph node-exporter.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:18:36.436 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Mar 04 16:18:36 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@node-exporter.a.service: Consumed 4.141s CPU time. 2024-03-04T16:18:36.737 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:36 smithi083 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a[59224]: level=info ts=2024-03-04T16:18:36.435Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-04T16:18:36.738 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:36 smithi083 podman[92704]: 2024-03-04 16:18:36.486671078 +0000 UTC m=+0.147235582 container died 3c6dc3779a0eabacd3304932ed447e199081f1e0bcebe25785ac102ad244df79 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:18:37.103 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:36 smithi083 podman[92704]: 2024-03-04 16:18:36.835814211 +0000 UTC m=+0.496378714 container cleanup 3c6dc3779a0eabacd3304932ed447e199081f1e0bcebe25785ac102ad244df79 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:18:37.103 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:36 smithi083 bash[92704]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a 2024-03-04T16:18:37.103 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:36 smithi083 bash[92731]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager.a" found: no such container 2024-03-04T16:18:37.376 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:37 smithi083 podman[92716]: 2024-03-04 16:18:37.103508319 +0000 UTC m=+0.621901752 container remove 3c6dc3779a0eabacd3304932ed447e199081f1e0bcebe25785ac102ad244df79 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:18:37.376 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:37 smithi083 podman[92716]: 2024-03-04 16:18:37.187559735 +0000 UTC m=+0.705953119 volume remove ee22d7beeb4d9010368196d51bffc47c4868fbc7b4a84f96d1e3b920a126d8e4 2024-03-04T16:18:37.376 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:37 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-04T16:18:37.376 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:37 smithi083 systemd[1]: Stopped Ceph alertmanager.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:18:37.376 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Mar 04 16:18:37 smithi083 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@alertmanager.a.service: Consumed 1.077s CPU time. 2024-03-04T16:19:11.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-04T16:19:13.484 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:19:13 smithi086 systemd[1]: Stopping Ceph node-exporter.b for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:19:13.484 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:19:13 smithi086 podman[77927]: 2024-03-04 16:19:13.311797541 +0000 UTC m=+0.115514358 container died b72dd80b2588b252d353cccc20e29588869ea0b17d66ae09f32cda7d0c36ba15 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T16:19:13.926 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:19:13 smithi086 podman[77927]: 2024-03-04 16:19:13.743223572 +0000 UTC m=+0.546940383 container cleanup b72dd80b2588b252d353cccc20e29588869ea0b17d66ae09f32cda7d0c36ba15 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T16:19:13.926 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:19:13 smithi086 bash[77927]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b 2024-03-04T16:19:14.234 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:19:14 smithi086 systemd[1]: Stopping Ceph grafana.a for 98ffad3c-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:19:14.234 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:19:13 smithi086 podman[77941]: 2024-03-04 16:19:13.926563375 +0000 UTC m=+0.615592982 container remove b72dd80b2588b252d353cccc20e29588869ea0b17d66ae09f32cda7d0c36ba15 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T16:19:14.234 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:19:13 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-04T16:19:14.234 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:19:13 smithi086 bash[77954]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-node-exporter.b" found: no such container 2024-03-04T16:19:14.234 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:19:13 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-04T16:19:14.234 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:19:13 smithi086 systemd[1]: Stopped Ceph node-exporter.b for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:19:14.235 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 04 16:19:13 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@node-exporter.b.service: Consumed 3.921s CPU time. 2024-03-04T16:19:14.547 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:19:14 smithi086 ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a[50952]: t=2024-03-04T16:19:14+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-03-04T16:19:14.548 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:19:14 smithi086 podman[78008]: 2024-03-04 16:19:14.284223693 +0000 UTC m=+0.126675352 container died fb754d58c53f55f4e9273328632647db3e8d690e8b4a88104fd63c9b106db0d7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, version=8.5, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, description=Ceph Grafana Container, maintainer=Paul Cuzner , vendor=Red Hat, Inc., name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=) 2024-03-04T16:19:14.804 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:19:14 smithi086 podman[78008]: 2024-03-04 16:19:14.621836499 +0000 UTC m=+0.464288150 container cleanup fb754d58c53f55f4e9273328632647db3e8d690e8b4a88104fd63c9b106db0d7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , vcs-type=git, architecture=x86_64, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, vendor=Red Hat, Inc., name=ubi8, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, release=236.1648460182, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-03-04T16:19:14.804 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:19:14 smithi086 bash[78008]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a 2024-03-04T16:19:14.804 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:19:14 smithi086 podman[78022]: 2024-03-04 16:19:14.78105761 +0000 UTC m=+0.498116625 container remove fb754d58c53f55f4e9273328632647db3e8d690e8b4a88104fd63c9b106db0d7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana-a, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, vcs-type=git, release=236.1648460182, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-03-04T16:19:15.234 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:19:14 smithi086 bash[78033]: Error: no container with name or ID "ceph-98ffad3c-da3f-11ee-95c6-87774f69a715-grafana.a" found: no such container 2024-03-04T16:19:15.234 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:19:14 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-04T16:19:15.234 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:19:14 smithi086 systemd[1]: Stopped Ceph grafana.a for 98ffad3c-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:19:15.234 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 04 16:19:14 smithi086 systemd[1]: ceph-98ffad3c-da3f-11ee-95c6-87774f69a715@grafana.a.service: Consumed 4.125s CPU time. 2024-03-04T16:19:37.701 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T16:19:37.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T16:19:37.767 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-04T16:19:37.768 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581240/remote/smithi083/crash 2024-03-04T16:19:37.769 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/crash -- . 2024-03-04T16:19:37.815 INFO:teuthology.orchestra.run.smithi083.stderr:tar: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-04T16:19:37.816 INFO:teuthology.orchestra.run.smithi083.stderr:tar: Error is not recoverable: exiting now 2024-03-04T16:19:37.818 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581240/remote/smithi086/crash 2024-03-04T16:19:37.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/crash -- . 2024-03-04T16:19:37.901 INFO:teuthology.orchestra.run.smithi086.stderr:tar: /var/lib/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-04T16:19:37.902 INFO:teuthology.orchestra.run.smithi086.stderr:tar: Error is not recoverable: exiting now 2024-03-04T16:19:37.903 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-04T16:19:37.903 DEBUG:teuthology.orchestra.run.smithi083:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-04T16:19:37.934 INFO:teuthology.orchestra.run.smithi083.stdout:2024-03-04T15:57:30.221569+0000 mon.a (mon.0) 165 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-04T16:19:37.934 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-04T16:19:37.935 DEBUG:teuthology.orchestra.run.smithi083:> sudo egrep '\[SEC\]' /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-04T16:19:38.009 DEBUG:teuthology.orchestra.run.smithi083:> sudo egrep '\[ERR\]' /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-04T16:19:38.078 DEBUG:teuthology.orchestra.run.smithi083:> sudo egrep '\[WRN\]' /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-04T16:19:38.146 INFO:teuthology.orchestra.run.smithi083.stdout:2024-03-04T15:57:30.221569+0000 mon.a (mon.0) 165 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-04T16:19:38.146 INFO:tasks.cephadm:Compressing logs... 2024-03-04T16:19:38.147 DEBUG:teuthology.orchestra.run.smithi083:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T16:19:38.188 DEBUG:teuthology.orchestra.run.smithi086:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T16:19:38.214 INFO:teuthology.orchestra.run.smithi083.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-04T16:19:38.214 INFO:teuthology.orchestra.run.smithi083.stderr:: No such file or directory 2024-03-04T16:19:38.215 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-mon.a.log 2024-03-04T16:19:38.215 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-client.rgw.smpl.smithi083.hqmkpf.log 2024-03-04T16:19:38.215 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-04T16:19:38.215 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-client.rgw.smpl.smithi083.hqmkpf.log: gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.1.log 2024-03-04T16:19:38.215 INFO:teuthology.orchestra.run.smithi083.stderr: 76.9% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-client.rgw.smpl.smithi083.hqmkpf.log.gz 2024-03-04T16:19:38.215 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.3.log 2024-03-04T16:19:38.215 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-mgr.y.log 2024-03-04T16:19:38.215 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.1.log: /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-client.rgw.foo.smithi083.szjzkw.log 2024-03-04T16:19:38.217 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/tcmu-runner.log 2024-03-04T16:19:38.217 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-mgr.y.log: /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-client.rgw.foo.smithi083.szjzkw.log: 76.8% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-client.rgw.foo.smithi083.szjzkw.log.gz 2024-03-04T16:19:38.217 INFO:teuthology.orchestra.run.smithi086.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-04T16:19:38.218 INFO:teuthology.orchestra.run.smithi086.stderr:: No such file or directory 2024-03-04T16:19:38.218 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.6.log 2024-03-04T16:19:38.218 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose/var/log/ceph/cephadm.log: -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-04T16:19:38.218 INFO:teuthology.orchestra.run.smithi086.stderr:gzip/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.6.log: -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-client.rgw.smpl.smithi086.yqigbl.log 2024-03-04T16:19:38.218 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.4.log 2024-03-04T16:19:38.218 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.7.log 2024-03-04T16:19:38.221 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-client.rgw.smpl.smithi086.yqigbl.log: /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.5.log 2024-03-04T16:19:38.221 INFO:teuthology.orchestra.run.smithi086.stderr: 77.3% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-client.rgw.smpl.smithi086.yqigbl.log.gz 2024-03-04T16:19:38.221 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-client.rgw.foo.smithi086.ezumfo.log 2024-03-04T16:19:38.221 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-mgr.x.log 2024-03-04T16:19:38.221 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-client.rgw.foo.smithi086.ezumfo.log: gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-04T16:19:38.221 INFO:teuthology.orchestra.run.smithi086.stderr: 76.3% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-client.rgw.foo.smithi086.ezumfo.log.gz 2024-03-04T16:19:38.221 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-mgr.x.log: 90.5% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-04T16:19:38.222 INFO:teuthology.orchestra.run.smithi083.stderr: 94.2% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-04T16:19:38.222 INFO:teuthology.orchestra.run.smithi083.stderr: 91.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-04T16:19:38.223 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.log 2024-03-04T16:19:38.223 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-04T16:19:38.223 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.log: 85.9% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-04T16:19:38.224 INFO:teuthology.orchestra.run.smithi086.stderr: 87.6% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-04T16:19:38.226 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-mon.b.log 2024-03-04T16:19:38.226 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-volume.log: 88.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-04T16:19:38.227 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.0.log 2024-03-04T16:19:38.227 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/tcmu-runner.log: 84.9% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/tcmu-runner.log.gz 2024-03-04T16:19:38.228 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-04T16:19:38.228 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.log 2024-03-04T16:19:38.228 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.2.log 2024-03-04T16:19:38.228 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.log: 91.8% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-04T16:19:38.232 INFO:teuthology.orchestra.run.smithi083.stderr: 93.0% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-04T16:19:38.232 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-04T16:19:38.233 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-mon.c.log 2024-03-04T16:19:38.234 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-mon.b.log: 91.4% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-04T16:19:38.259 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-volume.log: /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-mon.c.log: 91.3% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-04T16:19:38.279 INFO:teuthology.orchestra.run.smithi086.stderr: 88.9% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-mgr.x.log.gz 2024-03-04T16:19:38.542 INFO:teuthology.orchestra.run.smithi086.stderr: 92.4% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-mon.b.log.gz 2024-03-04T16:19:38.568 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-mgr.y.log.gz 2024-03-04T16:19:38.760 INFO:teuthology.orchestra.run.smithi083.stderr: 92.1% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-mon.c.log.gz 2024-03-04T16:19:39.214 INFO:teuthology.orchestra.run.smithi086.stderr: 93.5% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.6.log.gz 2024-03-04T16:19:39.220 INFO:teuthology.orchestra.run.smithi083.stderr: 91.0% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-mon.a.log.gz 2024-03-04T16:19:39.337 INFO:teuthology.orchestra.run.smithi083.stderr: 93.6% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.2.log.gz 2024-03-04T16:19:39.521 INFO:teuthology.orchestra.run.smithi086.stderr: 93.6% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.5.log.gz 2024-03-04T16:19:39.697 INFO:teuthology.orchestra.run.smithi086.stderr: 93.8% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.7.log.gz 2024-03-04T16:19:39.800 INFO:teuthology.orchestra.run.smithi086.stderr: 93.8% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.4.log.gz 2024-03-04T16:19:39.802 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-04T16:19:39.802 INFO:teuthology.orchestra.run.smithi086.stderr:real 0m1.599s 2024-03-04T16:19:39.802 INFO:teuthology.orchestra.run.smithi086.stderr:user 0m5.548s 2024-03-04T16:19:39.802 INFO:teuthology.orchestra.run.smithi086.stderr:sys 0m0.243s 2024-03-04T16:19:39.812 INFO:teuthology.orchestra.run.smithi083.stderr: 93.7% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.3.log.gz 2024-03-04T16:19:39.838 INFO:teuthology.orchestra.run.smithi083.stderr: 93.6% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.0.log.gz 2024-03-04T16:19:39.893 INFO:teuthology.orchestra.run.smithi083.stderr: 93.7% -- replaced with /var/log/ceph/98ffad3c-da3f-11ee-95c6-87774f69a715/ceph-osd.1.log.gz 2024-03-04T16:19:39.895 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-04T16:19:39.896 INFO:teuthology.orchestra.run.smithi083.stderr:real 0m1.696s 2024-03-04T16:19:39.896 INFO:teuthology.orchestra.run.smithi083.stderr:user 0m7.622s 2024-03-04T16:19:39.896 INFO:teuthology.orchestra.run.smithi083.stderr:sys 0m0.282s 2024-03-04T16:19:39.896 INFO:tasks.cephadm:Archiving logs... 2024-03-04T16:19:39.896 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/log/ceph to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581240/remote/smithi083/log 2024-03-04T16:19:39.897 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T16:19:40.907 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581240/remote/smithi086/log 2024-03-04T16:19:40.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T16:19:41.748 INFO:tasks.cephadm:Removing cluster... 2024-03-04T16:19:41.749 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 --force 2024-03-04T16:19:42.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 98ffad3c-da3f-11ee-95c6-87774f69a715 --force 2024-03-04T16:19:42.321 INFO:tasks.cephadm:Removing cephadm ... 2024-03-04T16:19:42.322 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-04T16:19:42.336 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-04T16:19:42.351 INFO:tasks.cephadm:Teardown complete 2024-03-04T16:19:42.352 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-04T16:19:42.363 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-04T16:19:42.363 DEBUG:teuthology.orchestra.run.smithi083:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T16:19:42.379 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T16:19:42.391 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-03-04T16:19:42.395 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T16:19:42.395 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-03-04T16:19:42.395 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 101 -1978us[-1978us] +/- 134ms 2024-03-04T16:19:42.395 INFO:teuthology.orchestra.run.smithi083.stdout:^* hv02.front.sepia.ceph.com 3 7 377 102 -613us[ -586us] +/- 71ms 2024-03-04T16:19:42.395 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +1674us[+1674us] +/- 69ms 2024-03-04T16:19:42.395 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T16:19:42.404 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-03-04T16:19:42.408 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T16:19:42.409 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-03-04T16:19:42.409 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 102 -1938us[-1958us] +/- 134ms 2024-03-04T16:19:42.409 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -615us[ -633us] +/- 72ms 2024-03-04T16:19:42.409 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 +1695us[+1695us] +/- 69ms 2024-03-04T16:19:42.409 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T16:19:42.409 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-04T16:19:42.418 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-04T16:19:42.419 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-04T16:19:42.427 DEBUG:teuthology.orchestra.run.smithi083:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T16:19:42.516 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T16:19:42.659 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T16:19:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:42.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T16:19:42.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:42.721 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-04T16:19:42.731 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-04T16:19:42.740 INFO:teuthology.task.internal:Duration was 1718.067951 seconds 2024-03-04T16:19:42.741 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-04T16:19:42.752 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-04T16:19:42.752 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T16:19:42.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T16:19:42.796 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T16:19:42.804 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T16:19:43.280 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-04T16:19:43.280 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi083.front.sepia.ceph.com 2024-03-04T16:19:43.280 DEBUG:teuthology.orchestra.run.smithi083:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T16:19:43.301 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2024-03-04T16:19:43.353 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T16:19:43.383 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-04T16:19:43.384 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T16:19:43.386 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T16:19:43.453 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-04T16:19:43.453 DEBUG:teuthology.orchestra.run.smithi083:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T16:19:43.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T16:19:43.880 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-04T16:19:43.891 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-04T16:19:43.891 DEBUG:teuthology.orchestra.run.smithi083:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T16:19:43.918 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T16:19:43.946 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-04T16:19:43.955 DEBUG:teuthology.orchestra.run.smithi083:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T16:19:43.961 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T16:19:43.984 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = core 2024-03-04T16:19:44.012 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2024-03-04T16:19:44.050 DEBUG:teuthology.orchestra.run.smithi083:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T16:19:44.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:44.064 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T16:19:44.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:44.104 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-04T16:19:44.115 INFO:teuthology.task.internal:Transferring archived files... 2024-03-04T16:19:44.116 DEBUG:teuthology.misc:Transferring archived files from smithi083:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581240/remote/smithi083 2024-03-04T16:19:44.116 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T16:19:44.162 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581240/remote/smithi086 2024-03-04T16:19:44.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T16:19:44.206 INFO:teuthology.task.internal:Removing archive directory... 2024-03-04T16:19:44.206 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T16:19:44.209 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T16:19:44.254 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-04T16:19:44.266 INFO:teuthology.task.internal:Not uploading archives. 2024-03-04T16:19:44.266 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-04T16:19:44.321 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-04T16:19:44.321 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T16:19:44.323 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T16:19:44.335 INFO:teuthology.orchestra.run.smithi083.stdout: 266038 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 16:19 /home/ubuntu/cephtest 2024-03-04T16:19:44.337 INFO:teuthology.orchestra.run.smithi086.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 16:19 /home/ubuntu/cephtest 2024-03-04T16:19:44.339 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-04T16:19:44.348 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-04T16:19:44.403 INFO:teuthology.nuke:Checking targets against current locks 2024-03-04T16:19:44.433 DEBUG:teuthology.nuke:shortname: smithi083 2024-03-04T16:19:44.433 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T16:19:44.458 DEBUG:teuthology.nuke:shortname: smithi086 2024-03-04T16:19:44.458 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T16:19:44.461 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581240', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:42:33.936299', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBEMT0OhSFWXWUnXcQSt30R69b09/MyThuIpLwcGR3YBD9YdlReB110ouWA539I2tqCPsShVn+qCZP3vdyTOSQs='} 2024-03-04T16:19:44.483 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581240', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:42:33.938259', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD8xljKq9p545rXF3lnZ87pN5AxQ2dPNhjW/ghJCiH9Evyz3TTVYN5XcpAI3B7Ck+Kk0IOYt9e+LqpK3ksCGAUQ='} 2024-03-04T16:19:44.517 INFO:teuthology.orchestra.console.smithi083:Power off 2024-03-04T16:19:44.517 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T16:19:44.541 DEBUG:teuthology.orchestra.console.smithi083:power off output: Chassis Power Control: Down/Off 2024-03-04T16:19:44.541 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:19:44.555 INFO:teuthology.orchestra.console.smithi086:Power off 2024-03-04T16:19:44.555 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T16:19:44.565 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-03-04T16:19:44.578 DEBUG:teuthology.orchestra.console.smithi086:power off output: Chassis Power Control: Down/Off 2024-03-04T16:19:44.578 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:19:44.603 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-03-04T16:19:48.565 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:19:48.604 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:19:48.675 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-03-04T16:19:48.714 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-03-04T16:19:52.679 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:19:52.715 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:19:52.789 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-03-04T16:19:52.825 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is off 2024-03-04T16:19:52.925 INFO:teuthology.orchestra.console.smithi086:Power off completed 2024-03-04T16:19:56.794 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:19:56.904 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is off 2024-03-04T16:19:57.005 INFO:teuthology.orchestra.console.smithi083:Power off completed 2024-03-04T16:19:57.109 INFO:teuthology.run:Summary data: description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 1718.0679507255554 failure_reason: '"2024-03-04T15:57:30.221569+0000 mon.a (mon.0) 165 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log' owner: scheduled_tonay@teuthology success: false 2024-03-04T16:19:57.109 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T16:19:57.189 INFO:teuthology.run:FAIL