2024-04-14T13:18:49.205 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-14T13:18:49.205 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7645991 --description orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7645991/orig.config.yaml 2024-04-14T13:18:49.248 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-14T13:18:49.338 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7645991 branch: main description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} email: ceph-qa@ceph.com first_in_suite: false job_id: '7645991' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: a971fd5f5719038c70b817bafacd6d5400b1d888 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: a971fd5f5719038c70b817bafacd6d5400b1d888 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: main sha1: 05c9d46e7de5db2cb434c177a6bbfc56f472d09b owner: scheduled_teuthology@teuthology priority: 950 repo: https://git.ceph.com/ceph.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 6509 sha1: a971fd5f5719038c70b817bafacd6d5400b1d888 sleep_before_teardown: 0 subset: 21/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_05c9d46e7de5db2cb434c177a6bbfc56f472d09b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 05c9d46e7de5db2cb434c177a6bbfc56f472d09b targets: smithi008.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIXjAFp/v4W4GpwJgDxyfDEBzUFsBxOr07eZaob/n4rP5YVh6fLBGfDMAniPdFOdWaf58tdvIwtvmcQ7l4UmNTA= smithi012.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIF6RKBD8smBvL6BjGpnEPPv3WDege88/LB1Swg7VaJ7JpEvMVNVwmrhlV9DQ7UXvaPY4PmD6/47087f32bS5+4= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.wait_for_service: service: mds.foofs - cephadm.shell: host.a: - ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2 - ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done - vip.exec: host.a: - mkdir /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done - echo test > /mnt/foo/testfile - sync - parallel: - upgrade-tasks - workload-tasks - vip.exec: host.a: - umount /mnt/foo - cephadm.shell: host.a: - ceph nfs cluster ls | grep foo - ceph nfs export ls foo --detailed - rados -p .nfs --all ls - - cephadm.shell: host.a: - 'set -ex [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-08_20:08:14 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph mgr module enable nfs --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - cephadm.wait_for_service: service: nfs.foo user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - exec: host.a: - cd /mnt/foo && dbench 5 -t 600 || true - umount /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done - cd /mnt/foo && dbench 5 -t 5 2024-04-14T13:18:49.338 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_05c9d46e7de5db2cb434c177a6bbfc56f472d09b/qa; will attempt to use it 2024-04-14T13:18:49.339 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_05c9d46e7de5db2cb434c177a6bbfc56f472d09b/qa/tasks 2024-04-14T13:18:49.339 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-14T13:18:49.342 INFO:teuthology.task.internal:Checking packages... 2024-04-14T13:18:49.367 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a971fd5f5719038c70b817bafacd6d5400b1d888' 2024-04-14T13:18:49.367 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-14T13:18:49.367 INFO:teuthology.packaging:ref: None 2024-04-14T13:18:49.367 INFO:teuthology.packaging:tag: None 2024-04-14T13:18:49.367 INFO:teuthology.packaging:branch: main 2024-04-14T13:18:49.367 INFO:teuthology.packaging:sha1: a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T13:18:49.367 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-04-14T13:18:49.589 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3080.gc2d8c315 2024-04-14T13:18:49.591 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-14T13:18:49.598 INFO:teuthology.task.internal:no buildpackages task found 2024-04-14T13:18:49.598 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-14T13:18:49.605 INFO:teuthology.task.internal:Saving configuration 2024-04-14T13:18:49.619 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-14T13:18:49.625 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-14T13:18:49.647 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi008.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7645991', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 13:14:04.478500', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIXjAFp/v4W4GpwJgDxyfDEBzUFsBxOr07eZaob/n4rP5YVh6fLBGfDMAniPdFOdWaf58tdvIwtvmcQ7l4UmNTA='} 2024-04-14T13:18:49.668 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7645991', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 13:14:04.479663', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIF6RKBD8smBvL6BjGpnEPPv3WDege88/LB1Swg7VaJ7JpEvMVNVwmrhlV9DQ7UXvaPY4PmD6/47087f32bS5+4='} 2024-04-14T13:18:49.668 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-14T13:18:49.676 INFO:teuthology.task.internal:roles: ubuntu@smithi008.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-14T13:18:49.676 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-04-14T13:18:49.676 INFO:teuthology.run_tasks:Running task console_log... 2024-04-14T13:18:49.748 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7d8af23f70>, signals=[15]) 2024-04-14T13:18:49.748 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-14T13:18:49.755 INFO:teuthology.task.internal:Opening connections... 2024-04-14T13:18:49.756 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi008.front.sepia.ceph.com 2024-04-14T13:18:49.757 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T13:18:49.861 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi012.front.sepia.ceph.com 2024-04-14T13:18:49.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T13:18:49.964 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-14T13:18:49.970 DEBUG:teuthology.orchestra.run.smithi008:> uname -m 2024-04-14T13:18:49.989 INFO:teuthology.orchestra.run.smithi008.stdout:x86_64 2024-04-14T13:18:49.989 DEBUG:teuthology.orchestra.run.smithi008:> cat /etc/os-release 2024-04-14T13:18:50.046 INFO:teuthology.orchestra.run.smithi008.stdout:NAME="CentOS Stream" 2024-04-14T13:18:50.046 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION="9" 2024-04-14T13:18:50.046 INFO:teuthology.orchestra.run.smithi008.stdout:ID="centos" 2024-04-14T13:18:50.046 INFO:teuthology.orchestra.run.smithi008.stdout:ID_LIKE="rhel fedora" 2024-04-14T13:18:50.046 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_ID="9" 2024-04-14T13:18:50.046 INFO:teuthology.orchestra.run.smithi008.stdout:PLATFORM_ID="platform:el9" 2024-04-14T13:18:50.046 INFO:teuthology.orchestra.run.smithi008.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-14T13:18:50.046 INFO:teuthology.orchestra.run.smithi008.stdout:ANSI_COLOR="0;31" 2024-04-14T13:18:50.046 INFO:teuthology.orchestra.run.smithi008.stdout:LOGO="fedora-logo-icon" 2024-04-14T13:18:50.046 INFO:teuthology.orchestra.run.smithi008.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-14T13:18:50.047 INFO:teuthology.orchestra.run.smithi008.stdout:HOME_URL="https://centos.org/" 2024-04-14T13:18:50.047 INFO:teuthology.orchestra.run.smithi008.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-14T13:18:50.047 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-14T13:18:50.047 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-14T13:18:50.047 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2024-04-14T13:18:50.080 DEBUG:teuthology.orchestra.run.smithi012:> uname -m 2024-04-14T13:18:50.097 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2024-04-14T13:18:50.097 DEBUG:teuthology.orchestra.run.smithi012:> cat /etc/os-release 2024-04-14T13:18:50.154 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="CentOS Stream" 2024-04-14T13:18:50.154 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="9" 2024-04-14T13:18:50.154 INFO:teuthology.orchestra.run.smithi012.stdout:ID="centos" 2024-04-14T13:18:50.154 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE="rhel fedora" 2024-04-14T13:18:50.154 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="9" 2024-04-14T13:18:50.154 INFO:teuthology.orchestra.run.smithi012.stdout:PLATFORM_ID="platform:el9" 2024-04-14T13:18:50.154 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-14T13:18:50.154 INFO:teuthology.orchestra.run.smithi012.stdout:ANSI_COLOR="0;31" 2024-04-14T13:18:50.154 INFO:teuthology.orchestra.run.smithi012.stdout:LOGO="fedora-logo-icon" 2024-04-14T13:18:50.154 INFO:teuthology.orchestra.run.smithi012.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-14T13:18:50.154 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://centos.org/" 2024-04-14T13:18:50.154 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-14T13:18:50.154 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-14T13:18:50.154 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-14T13:18:50.155 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2024-04-14T13:18:50.180 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-14T13:18:50.189 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-14T13:18:50.243 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-14T13:18:50.243 DEBUG:teuthology.orchestra.run.smithi008:> test '!' -e /home/ubuntu/cephtest 2024-04-14T13:18:50.245 DEBUG:teuthology.orchestra.run.smithi012:> test '!' -e /home/ubuntu/cephtest 2024-04-14T13:18:50.261 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-14T13:18:50.267 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-14T13:18:50.267 DEBUG:teuthology.orchestra.run.smithi008:> test -z $(ls -A /var/lib/ceph) 2024-04-14T13:18:50.303 DEBUG:teuthology.orchestra.run.smithi012:> test -z $(ls -A /var/lib/ceph) 2024-04-14T13:18:50.325 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-14T13:18:50.379 INFO:teuthology.run_tasks:Running task kernel... 2024-04-14T13:18:50.394 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-14T13:18:50.395 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-14T13:18:50.395 DEBUG:teuthology.orchestra.run.smithi008:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-14T13:18:50.395 DEBUG:teuthology.orchestra.run.smithi012:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-14T13:18:50.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T13:18:50.412 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-04-14T13:18:50.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T13:18:50.413 DEBUG:teuthology.orchestra.run.smithi008:> uname -r 2024-04-14T13:18:50.469 INFO:teuthology.orchestra.run.smithi012.stdout:5.14.0-361.el9.x86_64 2024-04-14T13:18:50.469 INFO:teuthology.task.kernel:Running kernel on smithi012: 5.14.0-361.el9.x86_64 2024-04-14T13:18:50.470 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum install -y kernel 2024-04-14T13:18:50.471 INFO:teuthology.orchestra.run.smithi008.stdout:5.14.0-361.el9.x86_64 2024-04-14T13:18:50.471 INFO:teuthology.task.kernel:Running kernel on smithi008: 5.14.0-361.el9.x86_64 2024-04-14T13:18:50.471 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum install -y kernel 2024-04-14T13:18:52.712 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - AppStream 12 MB/s | 8.1 MB 00:00 2024-04-14T13:18:52.715 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS-9 - AppStream 14 MB/s | 8.1 MB 00:00 2024-04-14T13:18:54.424 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS-9 - BaseOS 7.7 MB/s | 2.5 MB 00:00 2024-04-14T13:18:54.466 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.5 MB 00:00 2024-04-14T13:18:55.489 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-14T13:18:56.948 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.0 MB 00:02 2024-04-14T13:18:57.297 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-04-14T13:19:02.624 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.2 MB 00:01 2024-04-14T13:19:04.333 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-14T13:19:05.376 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-04-14T13:19:10.775 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-14T13:19:13.113 INFO:teuthology.orchestra.run.smithi012.stderr:Modular dependency problems: 2024-04-14T13:19:13.113 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-04-14T13:19:13.114 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 1: conflicting requests 2024-04-14T13:19:13.114 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.114 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 2: conflicting requests 2024-04-14T13:19:13.114 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.114 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 3: conflicting requests 2024-04-14T13:19:13.114 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.114 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 4: conflicting requests 2024-04-14T13:19:13.114 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.114 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 5: conflicting requests 2024-04-14T13:19:13.114 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.114 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 6: conflicting requests 2024-04-14T13:19:13.114 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.115 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 7: conflicting requests 2024-04-14T13:19:13.115 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.115 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 8: conflicting requests 2024-04-14T13:19:13.115 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.115 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 9: conflicting requests 2024-04-14T13:19:13.115 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.115 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 10: conflicting requests 2024-04-14T13:19:13.115 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.115 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 11: conflicting requests 2024-04-14T13:19:13.115 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.115 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 12: conflicting requests 2024-04-14T13:19:13.115 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.115 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 13: conflicting requests 2024-04-14T13:19:13.116 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.116 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 14: conflicting requests 2024-04-14T13:19:13.116 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.116 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 15: conflicting requests 2024-04-14T13:19:13.116 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.116 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 16: conflicting requests 2024-04-14T13:19:13.116 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.116 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 17: conflicting requests 2024-04-14T13:19:13.116 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.116 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 18: conflicting requests 2024-04-14T13:19:13.116 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.116 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 19: conflicting requests 2024-04-14T13:19:13.116 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.116 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 20: conflicting requests 2024-04-14T13:19:13.116 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.117 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 21: conflicting requests 2024-04-14T13:19:13.117 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.117 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 22: conflicting requests 2024-04-14T13:19:13.117 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.117 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 23: conflicting requests 2024-04-14T13:19:13.117 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.117 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 24: conflicting requests 2024-04-14T13:19:13.117 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.117 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 25: conflicting requests 2024-04-14T13:19:13.117 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.117 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 26: conflicting requests 2024-04-14T13:19:13.117 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.117 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 27: conflicting requests 2024-04-14T13:19:13.117 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.117 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 28: conflicting requests 2024-04-14T13:19:13.118 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.118 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 29: conflicting requests 2024-04-14T13:19:13.118 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.118 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 30: conflicting requests 2024-04-14T13:19:13.118 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.118 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 31: conflicting requests 2024-04-14T13:19:13.118 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.118 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 32: conflicting requests 2024-04-14T13:19:13.118 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.118 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 33: conflicting requests 2024-04-14T13:19:13.118 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.118 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 34: conflicting requests 2024-04-14T13:19:13.118 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.119 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 35: conflicting requests 2024-04-14T13:19:13.119 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.119 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 36: conflicting requests 2024-04-14T13:19:13.119 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.119 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 37: conflicting requests 2024-04-14T13:19:13.119 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.119 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.119 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.119 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T13:19:13.119 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.119 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.119 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.119 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.120 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 39: conflicting requests 2024-04-14T13:19:13.120 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.120 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.120 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.120 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.120 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.120 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.120 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T13:19:13.120 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.120 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.120 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.120 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.120 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T13:19:13.120 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.121 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.121 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.121 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.121 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T13:19:13.121 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.121 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.121 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.121 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.121 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T13:19:13.121 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.121 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.121 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.121 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.121 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T13:19:13.121 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.122 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.122 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.122 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.122 INFO:teuthology.orchestra.run.smithi012.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T13:19:13.122 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.122 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.122 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.122 INFO:teuthology.orchestra.run.smithi012.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T13:19:13.549 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-14T13:19:13.549 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-14T13:19:13.647 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-04-14T13:19:13.656 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-04-14T13:19:13.656 INFO:teuthology.orchestra.run.smithi012.stdout: Package Architecture Version Repository Size 2024-04-14T13:19:13.657 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-04-14T13:19:13.657 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-04-14T13:19:13.657 INFO:teuthology.orchestra.run.smithi012.stdout: kernel x86_64 5.14.0-435.el9 baseos 214 k 2024-04-14T13:19:13.657 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-04-14T13:19:13.657 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-core x86_64 5.14.0-435.el9 baseos 15 M 2024-04-14T13:19:13.657 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules x86_64 5.14.0-435.el9 baseos 34 M 2024-04-14T13:19:13.657 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-core x86_64 5.14.0-435.el9 baseos 28 M 2024-04-14T13:19:13.657 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:19:13.657 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-04-14T13:19:13.657 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-04-14T13:19:13.657 INFO:teuthology.orchestra.run.smithi012.stdout:Install 4 Packages 2024-04-14T13:19:13.657 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:19:13.660 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 77 M 2024-04-14T13:19:13.660 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 123 M 2024-04-14T13:19:13.660 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-04-14T13:19:14.035 INFO:teuthology.orchestra.run.smithi012.stdout:(1/4): kernel-5.14.0-435.el9.x86_64.rpm 997 kB/s | 214 kB 00:00 2024-04-14T13:19:14.668 INFO:teuthology.orchestra.run.smithi012.stdout:(2/4): kernel-core-5.14.0-435.el9.x86_64.rpm 18 MB/s | 15 MB 00:00 2024-04-14T13:19:15.352 INFO:teuthology.orchestra.run.smithi012.stdout:(3/4): kernel-modules-5.14.0-435.el9.x86_64.rpm 22 MB/s | 34 MB 00:01 2024-04-14T13:19:16.460 INFO:teuthology.orchestra.run.smithi012.stdout:(4/4): kernel-modules-core-5.14.0-435.el9.x86_6 11 MB/s | 28 MB 00:02 2024-04-14T13:19:16.461 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-04-14T13:19:16.461 INFO:teuthology.orchestra.run.smithi012.stdout:Total 27 MB/s | 77 MB 00:02 2024-04-14T13:19:17.159 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-04-14T13:19:17.578 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-04-14T13:19:17.578 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-04-14T13:19:18.632 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-04-14T13:19:18.633 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-04-14T13:19:20.922 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - AppStream 848 kB/s | 19 MB 00:22 2024-04-14T13:19:21.246 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-04-14T13:19:21.938 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-modules-core-5.14.0-435.el9.x86_64 1/4 2024-04-14T13:19:22.027 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T13:19:24.915 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T13:19:25.048 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T13:19:26.123 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.2 MB 00:01 2024-04-14T13:19:27.757 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-04-14T13:19:29.283 INFO:teuthology.orchestra.run.smithi008.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-04-14T13:19:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T13:19:30.174 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:19:34.634 INFO:teuthology.orchestra.run.smithi008.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-04-14T13:19:35.270 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:19:37.002 INFO:teuthology.orchestra.run.smithi008.stderr:Modular dependency problems: 2024-04-14T13:19:37.002 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-14T13:19:37.002 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 1: conflicting requests 2024-04-14T13:19:37.002 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.003 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 2: conflicting requests 2024-04-14T13:19:37.003 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.003 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 3: conflicting requests 2024-04-14T13:19:37.003 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.003 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 4: conflicting requests 2024-04-14T13:19:37.003 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.003 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 5: conflicting requests 2024-04-14T13:19:37.003 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.003 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 6: conflicting requests 2024-04-14T13:19:37.003 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.003 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 7: conflicting requests 2024-04-14T13:19:37.003 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.003 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 8: conflicting requests 2024-04-14T13:19:37.004 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.004 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 9: conflicting requests 2024-04-14T13:19:37.004 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.004 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 10: conflicting requests 2024-04-14T13:19:37.004 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.004 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 11: conflicting requests 2024-04-14T13:19:37.004 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.004 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 12: conflicting requests 2024-04-14T13:19:37.004 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.004 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 13: conflicting requests 2024-04-14T13:19:37.004 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.004 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 14: conflicting requests 2024-04-14T13:19:37.004 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.004 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 15: conflicting requests 2024-04-14T13:19:37.004 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.005 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 16: conflicting requests 2024-04-14T13:19:37.005 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.005 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 17: conflicting requests 2024-04-14T13:19:37.005 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.005 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 18: conflicting requests 2024-04-14T13:19:37.005 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.005 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 19: conflicting requests 2024-04-14T13:19:37.005 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.005 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 20: conflicting requests 2024-04-14T13:19:37.005 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.005 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 21: conflicting requests 2024-04-14T13:19:37.005 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.005 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 22: conflicting requests 2024-04-14T13:19:37.006 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.006 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 23: conflicting requests 2024-04-14T13:19:37.006 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.006 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 24: conflicting requests 2024-04-14T13:19:37.006 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.006 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 25: conflicting requests 2024-04-14T13:19:37.006 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.006 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 26: conflicting requests 2024-04-14T13:19:37.006 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.006 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 27: conflicting requests 2024-04-14T13:19:37.006 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.006 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 28: conflicting requests 2024-04-14T13:19:37.006 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.006 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 29: conflicting requests 2024-04-14T13:19:37.006 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.007 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 30: conflicting requests 2024-04-14T13:19:37.007 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.007 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 31: conflicting requests 2024-04-14T13:19:37.007 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.007 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 32: conflicting requests 2024-04-14T13:19:37.007 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.007 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 33: conflicting requests 2024-04-14T13:19:37.007 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.007 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 34: conflicting requests 2024-04-14T13:19:37.007 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.007 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 35: conflicting requests 2024-04-14T13:19:37.007 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.007 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 36: conflicting requests 2024-04-14T13:19:37.007 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.008 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 37: conflicting requests 2024-04-14T13:19:37.008 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.008 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.008 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.008 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T13:19:37.008 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.008 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.008 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.008 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.008 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 39: conflicting requests 2024-04-14T13:19:37.008 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.008 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.008 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.008 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.009 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.009 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.009 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T13:19:37.009 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.009 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.009 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.009 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.009 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T13:19:37.009 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.009 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.009 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.009 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.009 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T13:19:37.009 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.009 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.010 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.010 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.010 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T13:19:37.010 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.010 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.010 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.010 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.010 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T13:19:37.010 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.010 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.010 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.010 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.010 INFO:teuthology.orchestra.run.smithi008.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T13:19:37.010 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.011 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.011 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.011 INFO:teuthology.orchestra.run.smithi008.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T13:19:37.358 INFO:teuthology.orchestra.run.smithi008.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-14T13:19:37.358 INFO:teuthology.orchestra.run.smithi008.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-14T13:19:37.465 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-04-14T13:19:37.472 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-04-14T13:19:37.472 INFO:teuthology.orchestra.run.smithi008.stdout: Package Architecture Version Repository Size 2024-04-14T13:19:37.472 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-04-14T13:19:37.473 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-04-14T13:19:37.473 INFO:teuthology.orchestra.run.smithi008.stdout: kernel x86_64 5.14.0-435.el9 baseos 214 k 2024-04-14T13:19:37.473 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-04-14T13:19:37.473 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-core x86_64 5.14.0-435.el9 baseos 15 M 2024-04-14T13:19:37.473 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-modules x86_64 5.14.0-435.el9 baseos 34 M 2024-04-14T13:19:37.473 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-modules-core x86_64 5.14.0-435.el9 baseos 28 M 2024-04-14T13:19:37.473 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:19:37.473 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-04-14T13:19:37.473 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-04-14T13:19:37.473 INFO:teuthology.orchestra.run.smithi008.stdout:Install 4 Packages 2024-04-14T13:19:37.473 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:19:37.474 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 77 M 2024-04-14T13:19:37.474 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 123 M 2024-04-14T13:19:37.474 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-04-14T13:19:37.951 INFO:teuthology.orchestra.run.smithi008.stdout:(1/4): kernel-5.14.0-435.el9.x86_64.rpm 719 kB/s | 214 kB 00:00 2024-04-14T13:19:38.577 INFO:teuthology.orchestra.run.smithi008.stdout:(2/4): kernel-core-5.14.0-435.el9.x86_64.rpm 16 MB/s | 15 MB 00:00 2024-04-14T13:19:39.894 INFO:teuthology.orchestra.run.smithi008.stdout:(3/4): kernel-modules-core-5.14.0-435.el9.x86_6 14 MB/s | 28 MB 00:01 2024-04-14T13:19:40.211 INFO:teuthology.orchestra.run.smithi008.stdout:(4/4): kernel-modules-5.14.0-435.el9.x86_64.rpm 13 MB/s | 34 MB 00:02 2024-04-14T13:19:40.213 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-04-14T13:19:40.214 INFO:teuthology.orchestra.run.smithi008.stdout:Total 28 MB/s | 77 MB 00:02 2024-04-14T13:19:40.920 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-04-14T13:19:41.364 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-04-14T13:19:41.364 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-04-14T13:19:42.406 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-04-14T13:19:42.407 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-04-14T13:19:45.124 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-04-14T13:19:45.745 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : kernel-modules-core-5.14.0-435.el9.x86_64 1/4 2024-04-14T13:19:45.820 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T13:19:48.749 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T13:19:48.892 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T13:19:53.976 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T13:19:54.019 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : kernel-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:19:59.114 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:20:05.988 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:20:05.989 INFO:teuthology.orchestra.run.smithi012.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-14T13:20:05.989 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:20:06.036 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:20:06.376 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:20:06.376 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-5.14.0-435.el9.x86_64 1/4 2024-04-14T13:20:06.376 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T13:20:06.376 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T13:20:06.849 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:20:06.849 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:20:06.849 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-04-14T13:20:06.850 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-5.14.0-435.el9.x86_64 2024-04-14T13:20:06.850 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-core-5.14.0-435.el9.x86_64 2024-04-14T13:20:06.850 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-5.14.0-435.el9.x86_64 2024-04-14T13:20:06.850 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-core-5.14.0-435.el9.x86_64 2024-04-14T13:20:06.850 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:20:06.850 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-04-14T13:20:07.190 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel | sort -rV | head -n 1 2024-04-14T13:20:07.297 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-14T13:20:07.297 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-14T13:20:07.297 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-435.el9.x86_64 2024-04-14T13:20:07.297 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-14T13:20:07.297 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-14T13:20:07.298 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-04-14T13:20:07.298 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi012.front.sepia.ceph.com, path=None, version=distro) 2024-04-14T13:20:07.298 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel | sort -rV | head -n 1 2024-04-14T13:20:07.381 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-14T13:20:07.382 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-14T13:20:07.382 DEBUG:teuthology.orchestra.run.smithi012:> sudo rpm -qi grub2-tools 2024-04-14T13:20:07.420 INFO:teuthology.orchestra.run.smithi012.stdout:Name : grub2-tools 2024-04-14T13:20:07.420 INFO:teuthology.orchestra.run.smithi012.stdout:Epoch : 1 2024-04-14T13:20:07.421 INFO:teuthology.orchestra.run.smithi012.stdout:Version : 2.06 2024-04-14T13:20:07.421 INFO:teuthology.orchestra.run.smithi012.stdout:Release : 68.el9 2024-04-14T13:20:07.421 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: x86_64 2024-04-14T13:20:07.421 INFO:teuthology.orchestra.run.smithi012.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-14T13:20:07.421 INFO:teuthology.orchestra.run.smithi012.stdout:Group : Unspecified 2024-04-14T13:20:07.421 INFO:teuthology.orchestra.run.smithi012.stdout:Size : 8264254 2024-04-14T13:20:07.421 INFO:teuthology.orchestra.run.smithi012.stdout:License : GPLv3+ 2024-04-14T13:20:07.421 INFO:teuthology.orchestra.run.smithi012.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-14T13:20:07.421 INFO:teuthology.orchestra.run.smithi012.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-14T13:20:07.421 INFO:teuthology.orchestra.run.smithi012.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-14T13:20:07.421 INFO:teuthology.orchestra.run.smithi012.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-14T13:20:07.421 INFO:teuthology.orchestra.run.smithi012.stdout:Packager : builder@centos.org 2024-04-14T13:20:07.421 INFO:teuthology.orchestra.run.smithi012.stdout:Vendor : CentOS 2024-04-14T13:20:07.421 INFO:teuthology.orchestra.run.smithi012.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-14T13:20:07.422 INFO:teuthology.orchestra.run.smithi012.stdout:Summary : Support tools for GRUB. 2024-04-14T13:20:07.422 INFO:teuthology.orchestra.run.smithi012.stdout:Description : 2024-04-14T13:20:07.422 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:20:07.422 INFO:teuthology.orchestra.run.smithi012.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-14T13:20:07.422 INFO:teuthology.orchestra.run.smithi012.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-14T13:20:07.422 INFO:teuthology.orchestra.run.smithi012.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-14T13:20:07.422 INFO:teuthology.orchestra.run.smithi012.stdout:hardware devices. 2024-04-14T13:20:07.422 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:20:07.422 INFO:teuthology.orchestra.run.smithi012.stdout:This subpackage provides tools for support of all platforms. 2024-04-14T13:20:07.423 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-14T13:20:07.423 INFO:teuthology.task.kernel:Updating grub on smithi012 to boot 5.14.0-435.el9.x86_64 2024-04-14T13:20:07.423 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-14T13:20:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:Generating grub configuration file ... 2024-04-14T13:20:09.350 INFO:teuthology.orchestra.run.smithi012.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-14T13:20:09.378 INFO:teuthology.orchestra.run.smithi012.stderr:done 2024-04-14T13:20:09.380 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2024-04-14T13:20:09.395 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.Bdg6vyP04E 2024-04-14T13:20:09.396 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Bdg6vyP04E 2024-04-14T13:20:09.469 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.Bdg6vyP04E 2024-04-14T13:20:09.618 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.Bdg6vyP04E is 6KB 2024-04-14T13:20:09.667 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.Bdg6vyP04E 2024-04-14T13:20:09.682 DEBUG:teuthology.orchestra.run.smithi012:> sudo /bin/ls /boot/loader/entries || true 2024-04-14T13:20:09.760 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-14T13:20:09.760 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-14T13:20:09.760 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-14T13:20:09.760 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64.conf 2024-04-14T13:20:09.761 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64 2024-04-14T13:20:09.918 DEBUG:teuthology.orchestra.run.smithi012:> sudo shutdown -r now 2024-04-14T13:20:29.907 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:20:29.907 INFO:teuthology.orchestra.run.smithi008.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-14T13:20:29.907 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:20:29.954 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:20:30.332 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:20:30.332 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : kernel-5.14.0-435.el9.x86_64 1/4 2024-04-14T13:20:30.332 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T13:20:30.332 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T13:20:30.756 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:20:30.756 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:20:30.756 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-04-14T13:20:30.756 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-5.14.0-435.el9.x86_64 2024-04-14T13:20:30.756 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-core-5.14.0-435.el9.x86_64 2024-04-14T13:20:30.756 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-modules-5.14.0-435.el9.x86_64 2024-04-14T13:20:30.756 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-modules-core-5.14.0-435.el9.x86_64 2024-04-14T13:20:30.756 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:20:30.756 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-04-14T13:20:31.104 DEBUG:teuthology.orchestra.run.smithi008:> rpm -q kernel | sort -rV | head -n 1 2024-04-14T13:20:31.205 INFO:teuthology.orchestra.run.smithi008.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-14T13:20:31.206 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-14T13:20:31.207 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-435.el9.x86_64 2024-04-14T13:20:31.207 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-14T13:20:31.207 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-14T13:20:31.207 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-04-14T13:20:31.207 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi008.front.sepia.ceph.com, path=None, version=distro) 2024-04-14T13:20:31.207 DEBUG:teuthology.orchestra.run.smithi008:> rpm -q kernel | sort -rV | head -n 1 2024-04-14T13:20:31.291 INFO:teuthology.orchestra.run.smithi008.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-14T13:20:31.292 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-14T13:20:31.292 DEBUG:teuthology.orchestra.run.smithi008:> sudo rpm -qi grub2-tools 2024-04-14T13:20:31.329 INFO:teuthology.orchestra.run.smithi008.stdout:Name : grub2-tools 2024-04-14T13:20:31.329 INFO:teuthology.orchestra.run.smithi008.stdout:Epoch : 1 2024-04-14T13:20:31.329 INFO:teuthology.orchestra.run.smithi008.stdout:Version : 2.06 2024-04-14T13:20:31.329 INFO:teuthology.orchestra.run.smithi008.stdout:Release : 68.el9 2024-04-14T13:20:31.330 INFO:teuthology.orchestra.run.smithi008.stdout:Architecture: x86_64 2024-04-14T13:20:31.330 INFO:teuthology.orchestra.run.smithi008.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-14T13:20:31.330 INFO:teuthology.orchestra.run.smithi008.stdout:Group : Unspecified 2024-04-14T13:20:31.330 INFO:teuthology.orchestra.run.smithi008.stdout:Size : 8264254 2024-04-14T13:20:31.330 INFO:teuthology.orchestra.run.smithi008.stdout:License : GPLv3+ 2024-04-14T13:20:31.330 INFO:teuthology.orchestra.run.smithi008.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-14T13:20:31.330 INFO:teuthology.orchestra.run.smithi008.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-14T13:20:31.330 INFO:teuthology.orchestra.run.smithi008.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-14T13:20:31.330 INFO:teuthology.orchestra.run.smithi008.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-14T13:20:31.330 INFO:teuthology.orchestra.run.smithi008.stdout:Packager : builder@centos.org 2024-04-14T13:20:31.330 INFO:teuthology.orchestra.run.smithi008.stdout:Vendor : CentOS 2024-04-14T13:20:31.330 INFO:teuthology.orchestra.run.smithi008.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-14T13:20:31.330 INFO:teuthology.orchestra.run.smithi008.stdout:Summary : Support tools for GRUB. 2024-04-14T13:20:31.330 INFO:teuthology.orchestra.run.smithi008.stdout:Description : 2024-04-14T13:20:31.331 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:20:31.331 INFO:teuthology.orchestra.run.smithi008.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-14T13:20:31.331 INFO:teuthology.orchestra.run.smithi008.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-14T13:20:31.331 INFO:teuthology.orchestra.run.smithi008.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-14T13:20:31.331 INFO:teuthology.orchestra.run.smithi008.stdout:hardware devices. 2024-04-14T13:20:31.331 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:20:31.331 INFO:teuthology.orchestra.run.smithi008.stdout:This subpackage provides tools for support of all platforms. 2024-04-14T13:20:31.332 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-14T13:20:31.332 INFO:teuthology.task.kernel:Updating grub on smithi008 to boot 5.14.0-435.el9.x86_64 2024-04-14T13:20:31.332 DEBUG:teuthology.orchestra.run.smithi008:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-14T13:20:31.896 INFO:teuthology.orchestra.run.smithi008.stderr:Generating grub configuration file ... 2024-04-14T13:20:33.294 INFO:teuthology.orchestra.run.smithi008.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-14T13:20:33.331 INFO:teuthology.orchestra.run.smithi008.stderr:done 2024-04-14T13:20:33.333 DEBUG:teuthology.orchestra.run.smithi008:> mktemp 2024-04-14T13:20:33.348 INFO:teuthology.orchestra.run.smithi008.stdout:/tmp/tmp.rsvmvBOzHW 2024-04-14T13:20:33.349 DEBUG:teuthology.orchestra.run.smithi008:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rsvmvBOzHW 2024-04-14T13:20:33.421 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 0666 /tmp/tmp.rsvmvBOzHW 2024-04-14T13:20:33.565 DEBUG:teuthology.orchestra.remote:smithi008:/tmp/tmp.rsvmvBOzHW is 6KB 2024-04-14T13:20:33.616 DEBUG:teuthology.orchestra.run.smithi008:> rm -fr /tmp/tmp.rsvmvBOzHW 2024-04-14T13:20:33.630 DEBUG:teuthology.orchestra.run.smithi008:> sudo /bin/ls /boot/loader/entries || true 2024-04-14T13:20:33.702 INFO:teuthology.orchestra.run.smithi008.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-14T13:20:33.702 INFO:teuthology.orchestra.run.smithi008.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-14T13:20:33.702 INFO:teuthology.orchestra.run.smithi008.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-14T13:20:33.702 INFO:teuthology.orchestra.run.smithi008.stdout:15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64.conf 2024-04-14T13:20:33.703 DEBUG:teuthology.orchestra.run.smithi008:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64 2024-04-14T13:20:33.885 DEBUG:teuthology.orchestra.run.smithi008:> sudo shutdown -r now 2024-04-14T13:20:39.923 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-04-14T13:20:39.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-04-14T13:20:39.925 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T13:20:58.398 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-04-14T13:21:03.890 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-04-14T13:21:03.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-04-14T13:21:03.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T13:21:07.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-04-14T13:21:07.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T13:21:22.211 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.8 2024-04-14T13:21:25.790 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-04-14T13:21:31.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-04-14T13:21:31.215 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T13:21:34.270 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.8 2024-04-14T13:21:37.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-04-14T13:21:37.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T13:21:39.022 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-04-14T13:21:39.453 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-04-14T13:21:39.453 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-435.el9.x86_64"... 2024-04-14T13:21:39.454 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-04-14T13:21:39.470 INFO:teuthology.orchestra.run.smithi012.stdout:5.14.0-435.el9.x86_64 2024-04-14T13:21:39.470 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-435.el9.x86_64 vs 5.14.0-435.el9.x86_64 2024-04-14T13:21:39.470 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-14T13:21:39.470 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-14T13:21:40.471 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-14T13:21:40.471 DEBUG:teuthology.orchestra.run.smithi012:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-14T13:21:40.568 INFO:teuthology.orchestra.run.smithi012.stdout:ttyS1 2024-04-14T13:21:40.625 DEBUG:teuthology.parallel:result is None 2024-04-14T13:21:46.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-04-14T13:21:46.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T13:22:18.080 DEBUG:teuthology.orchestra.run.smithi008:> true 2024-04-14T13:22:18.433 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-04-14T13:22:18.433 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-435.el9.x86_64"... 2024-04-14T13:22:18.433 DEBUG:teuthology.orchestra.run.smithi008:> uname -r 2024-04-14T13:22:18.447 INFO:teuthology.orchestra.run.smithi008.stdout:5.14.0-435.el9.x86_64 2024-04-14T13:22:18.447 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-435.el9.x86_64 vs 5.14.0-435.el9.x86_64 2024-04-14T13:22:18.447 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-14T13:22:18.447 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-14T13:22:19.448 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-14T13:22:19.448 DEBUG:teuthology.orchestra.run.smithi008:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-14T13:22:19.540 INFO:teuthology.orchestra.run.smithi008.stdout:ttyS1 2024-04-14T13:22:19.576 DEBUG:teuthology.parallel:result is None 2024-04-14T13:22:19.577 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-14T13:22:19.585 INFO:teuthology.task.internal:Creating test directory... 2024-04-14T13:22:19.586 DEBUG:teuthology.orchestra.run.smithi008:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-14T13:22:19.588 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-14T13:22:19.608 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-14T13:22:19.614 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-14T13:22:19.621 INFO:teuthology.task.internal:Creating archive directory... 2024-04-14T13:22:19.621 DEBUG:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-14T13:22:19.643 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-14T13:22:19.676 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-14T13:22:19.683 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-14T13:22:19.684 DEBUG:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-14T13:22:19.705 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-14T13:22:19.740 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T13:22:19.753 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T13:22:19.755 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T13:22:19.767 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T13:22:19.769 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-14T13:22:19.776 INFO:teuthology.task.internal:Configuring sudo... 2024-04-14T13:22:19.776 DEBUG:teuthology.orchestra.run.smithi008:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-14T13:22:19.797 DEBUG:teuthology.orchestra.run.smithi012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-14T13:22:19.839 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-14T13:22:19.848 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-14T13:22:19.849 DEBUG:teuthology.orchestra.run.smithi008:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-14T13:22:19.866 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-14T13:22:19.895 DEBUG:teuthology.orchestra.run.smithi008:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T13:22:19.958 DEBUG:teuthology.orchestra.run.smithi008:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T13:22:20.014 DEBUG:teuthology.orchestra.run.smithi008:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T13:22:20.096 DEBUG:teuthology.orchestra.run.smithi008:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T13:22:20.137 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-14T13:22:20.138 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-14T13:22:20.206 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T13:22:20.248 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T13:22:20.304 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T13:22:20.385 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T13:22:20.436 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-14T13:22:20.436 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-14T13:22:20.507 DEBUG:teuthology.orchestra.run.smithi008:> sudo service rsyslog restart 2024-04-14T13:22:20.510 DEBUG:teuthology.orchestra.run.smithi012:> sudo service rsyslog restart 2024-04-14T13:22:20.566 INFO:teuthology.orchestra.run.smithi008.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T13:22:20.593 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T13:22:21.088 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-14T13:22:21.097 INFO:teuthology.task.internal:Starting timer... 2024-04-14T13:22:21.098 INFO:teuthology.run_tasks:Running task pcp... 2024-04-14T13:22:21.108 INFO:teuthology.run_tasks:Running task selinux... 2024-04-14T13:22:21.120 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-14T13:22:21.120 DEBUG:teuthology.orchestra.run.smithi008:> sudo service auditd rotate 2024-04-14T13:22:21.202 INFO:teuthology.orchestra.run.smithi008.stdout:Rotating logs: 2024-04-14T13:22:21.204 DEBUG:teuthology.orchestra.run.smithi012:> sudo service auditd rotate 2024-04-14T13:22:21.258 INFO:teuthology.orchestra.run.smithi012.stdout:Rotating logs: 2024-04-14T13:22:21.260 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-14T13:22:21.260 DEBUG:teuthology.orchestra.run.smithi008:> /usr/sbin/getenforce 2024-04-14T13:22:21.287 INFO:teuthology.orchestra.run.smithi008.stdout:Permissive 2024-04-14T13:22:21.287 DEBUG:teuthology.orchestra.run.smithi012:> /usr/sbin/getenforce 2024-04-14T13:22:21.338 INFO:teuthology.orchestra.run.smithi012.stdout:Permissive 2024-04-14T13:22:21.338 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi008.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi012.front.sepia.ceph.com': 'permissive'} 2024-04-14T13:22:21.339 DEBUG:teuthology.orchestra.run.smithi008:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-14T13:22:21.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T13:22:21.367 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-14T13:22:21.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T13:22:21.398 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-14T13:22:21.398 DEBUG:teuthology.orchestra.run.smithi008:> sudo /usr/sbin/setenforce permissive 2024-04-14T13:22:21.447 DEBUG:teuthology.orchestra.run.smithi012:> sudo /usr/sbin/setenforce permissive 2024-04-14T13:22:21.482 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-14T13:22:21.492 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-14T13:22:21.511 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-14T13:22:21.512 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi008.front.sepia.ceph.com,smithi012.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-14T13:26:53.224 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi008.front.sepia.ceph.com'), Remote(name='ubuntu@smithi012.front.sepia.ceph.com')] 2024-04-14T13:26:53.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-04-14T13:26:53.226 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T13:26:53.317 DEBUG:teuthology.orchestra.run.smithi008:> true 2024-04-14T13:26:53.396 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-04-14T13:26:53.397 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-04-14T13:26:53.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T13:26:53.486 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-04-14T13:26:53.566 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-04-14T13:26:53.566 INFO:teuthology.run_tasks:Running task clock... 2024-04-14T13:26:53.577 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-14T13:26:53.577 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-14T13:26:53.577 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T13:26:53.579 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-14T13:26:53.579 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T13:26:53.608 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-14T13:26:53.625 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-14T13:26:53.640 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-14T13:26:53.655 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: ntpd: command not found 2024-04-14T13:26:53.657 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-14T13:26:53.669 INFO:teuthology.orchestra.run.smithi008.stdout:506 Cannot talk to daemon 2024-04-14T13:26:53.685 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-14T13:26:53.688 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: ntpd: command not found 2024-04-14T13:26:53.701 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-14T13:26:53.702 INFO:teuthology.orchestra.run.smithi012.stdout:506 Cannot talk to daemon 2024-04-14T13:26:53.719 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-14T13:26:53.735 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-14T13:26:53.755 INFO:teuthology.orchestra.run.smithi008.stderr:bash: line 1: ntpq: command not found 2024-04-14T13:26:53.759 INFO:teuthology.orchestra.run.smithi008.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T13:26:53.759 INFO:teuthology.orchestra.run.smithi008.stdout:=============================================================================== 2024-04-14T13:26:53.759 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T13:26:53.759 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T13:26:53.759 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T13:26:53.759 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T13:26:53.786 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2024-04-14T13:26:53.789 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T13:26:53.789 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-04-14T13:26:53.790 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T13:26:53.790 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T13:26:53.790 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T13:26:53.790 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T13:26:53.790 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-14T13:26:53.891 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'a971fd5f5719038c70b817bafacd6d5400b1d888'} 2024-04-14T13:26:53.892 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-14T13:26:53.892 INFO:tasks.cephadm:Cluster fsid is a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:26:53.892 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-14T13:26:53.892 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-14T13:26:53.893 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi008': '172.21.15.8', 'mon.smithi012': '172.21.15.12'} 2024-04-14T13:26:53.893 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-14T13:26:53.893 DEBUG:teuthology.orchestra.run.smithi008:> sudo hostname $(hostname -s) 2024-04-14T13:26:53.921 DEBUG:teuthology.orchestra.run.smithi012:> sudo hostname $(hostname -s) 2024-04-14T13:26:53.948 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-14T13:26:53.949 DEBUG:teuthology.orchestra.run.smithi008:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-14T13:26:54.214 INFO:teuthology.orchestra.run.smithi008.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 14 13:26 /home/ubuntu/cephtest/cephadm 2024-04-14T13:26:54.214 DEBUG:teuthology.orchestra.run.smithi012:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-14T13:26:54.380 INFO:teuthology.orchestra.run.smithi012.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 14 13:26 /home/ubuntu/cephtest/cephadm 2024-04-14T13:26:54.380 DEBUG:teuthology.orchestra.run.smithi008:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-14T13:26:54.396 DEBUG:teuthology.orchestra.run.smithi012:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-14T13:26:54.420 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-14T13:26:54.420 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-14T13:26:54.439 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-14T13:26:55.325 INFO:teuthology.orchestra.run.smithi012.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-14T13:26:55.333 INFO:teuthology.orchestra.run.smithi008.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-14T13:27:18.813 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:27:18.814 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-14T13:27:18.814 INFO:teuthology.orchestra.run.smithi008.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-14T13:27:18.814 INFO:teuthology.orchestra.run.smithi008.stdout: "repo_digests": [ 2024-04-14T13:27:18.814 INFO:teuthology.orchestra.run.smithi008.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-14T13:27:18.814 INFO:teuthology.orchestra.run.smithi008.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-14T13:27:18.814 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-14T13:27:18.814 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:27:21.591 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-14T13:27:21.591 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-14T13:27:21.591 INFO:teuthology.orchestra.run.smithi012.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-14T13:27:21.592 INFO:teuthology.orchestra.run.smithi012.stdout: "repo_digests": [ 2024-04-14T13:27:21.592 INFO:teuthology.orchestra.run.smithi012.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-14T13:27:21.592 INFO:teuthology.orchestra.run.smithi012.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-14T13:27:21.592 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-14T13:27:21.592 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-14T13:27:21.616 DEBUG:teuthology.orchestra.run.smithi008:> sudo mkdir -p /etc/ceph 2024-04-14T13:27:21.650 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /etc/ceph 2024-04-14T13:27:21.680 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 777 /etc/ceph 2024-04-14T13:27:21.719 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 777 /etc/ceph 2024-04-14T13:27:21.746 INFO:tasks.cephadm:Writing seed config... 2024-04-14T13:27:21.748 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-14T13:27:21.748 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-14T13:27:21.748 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-14T13:27:21.748 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-14T13:27:21.748 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-14T13:27:21.748 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-14T13:27:21.748 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-14T13:27:21.748 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-14T13:27:21.749 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-14T13:27:21.749 DEBUG:teuthology.orchestra.run.smithi008:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-14T13:27:21.775 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a8b03f6a-fa62-11ee-bc8e-c7b262605968 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-14T13:27:21.776 DEBUG:teuthology.orchestra.run.smithi008:mon.smithi008> sudo journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@mon.smithi008.service 2024-04-14T13:27:21.817 INFO:tasks.cephadm:Bootstrapping... 2024-04-14T13:27:21.818 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.8 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-14T13:27:22.019 INFO:teuthology.orchestra.run.smithi008.stderr:-------------------------------------------------------------------------------- 2024-04-14T13:27:22.019 INFO:teuthology.orchestra.run.smithi008.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'a8b03f6a-fa62-11ee-bc8e-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.8', '--skip-admin-label'] 2024-04-14T13:27:22.043 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: 4.9.4-dev 2024-04-14T13:27:22.046 INFO:teuthology.orchestra.run.smithi008.stderr:Verifying podman|docker is present... 2024-04-14T13:27:22.067 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: 4.9.4-dev 2024-04-14T13:27:22.069 INFO:teuthology.orchestra.run.smithi008.stderr:Verifying lvm2 is present... 2024-04-14T13:27:22.069 INFO:teuthology.orchestra.run.smithi008.stderr:Verifying time synchronization is in place... 2024-04-14T13:27:22.075 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-14T13:27:22.081 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: inactive 2024-04-14T13:27:22.088 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: enabled 2024-04-14T13:27:22.094 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: active 2024-04-14T13:27:22.094 INFO:teuthology.orchestra.run.smithi008.stderr:Unit chronyd.service is enabled and running 2024-04-14T13:27:22.094 INFO:teuthology.orchestra.run.smithi008.stderr:Repeating the final host check... 2024-04-14T13:27:22.115 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: 4.9.4-dev 2024-04-14T13:27:22.117 INFO:teuthology.orchestra.run.smithi008.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-14T13:27:22.117 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl is present 2024-04-14T13:27:22.117 INFO:teuthology.orchestra.run.smithi008.stderr:lvcreate is present 2024-04-14T13:27:22.123 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-14T13:27:22.129 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: inactive 2024-04-14T13:27:22.135 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: enabled 2024-04-14T13:27:22.141 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: active 2024-04-14T13:27:22.142 INFO:teuthology.orchestra.run.smithi008.stderr:Unit chronyd.service is enabled and running 2024-04-14T13:27:22.142 INFO:teuthology.orchestra.run.smithi008.stderr:Host looks OK 2024-04-14T13:27:22.142 INFO:teuthology.orchestra.run.smithi008.stderr:Cluster fsid: a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:27:22.142 INFO:teuthology.orchestra.run.smithi008.stderr:Acquiring lock 140615991225744 on /run/cephadm/a8b03f6a-fa62-11ee-bc8e-c7b262605968.lock 2024-04-14T13:27:22.142 INFO:teuthology.orchestra.run.smithi008.stderr:Lock 140615991225744 acquired on /run/cephadm/a8b03f6a-fa62-11ee-bc8e-c7b262605968.lock 2024-04-14T13:27:22.143 INFO:teuthology.orchestra.run.smithi008.stderr:Verifying IP 172.21.15.8 port 3300 ... 2024-04-14T13:27:22.143 INFO:teuthology.orchestra.run.smithi008.stderr:Verifying IP 172.21.15.8 port 6789 ... 2024-04-14T13:27:22.143 INFO:teuthology.orchestra.run.smithi008.stderr:Base mon IP is 172.21.15.8, final addrv is [v2:172.21.15.8:3300,v1:172.21.15.8:6789] 2024-04-14T13:27:22.145 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.8 metric 100 2024-04-14T13:27:22.145 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.8 metric 100 2024-04-14T13:27:22.148 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-14T13:27:22.148 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-14T13:27:22.148 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-14T13:27:22.150 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-14T13:27:22.150 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-14T13:27:22.150 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-14T13:27:22.150 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-14T13:27:22.151 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: inet6 fe80::ae1f:6bff:fef8:25fe/64 scope link noprefixroute 2024-04-14T13:27:22.151 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-14T13:27:22.151 INFO:teuthology.orchestra.run.smithi008.stderr:Mon IP `172.21.15.8` is in CIDR network `172.21.0.0/20` 2024-04-14T13:27:22.151 INFO:teuthology.orchestra.run.smithi008.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-14T13:27:22.152 INFO:teuthology.orchestra.run.smithi008.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-14T13:27:22.256 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-14T13:27:22.611 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Getting image source signatures 2024-04-14T13:27:22.611 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-14T13:27:22.611 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-14T13:27:22.612 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-14T13:27:22.612 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-14T13:27:22.612 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-14T13:27:22.612 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-14T13:27:22.755 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Writing manifest to image destination 2024-04-14T13:27:22.908 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-14T13:27:23.321 INFO:teuthology.orchestra.run.smithi008.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-14T13:27:23.773 INFO:teuthology.orchestra.run.smithi008.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-14T13:27:23.773 INFO:teuthology.orchestra.run.smithi008.stderr:Extracting ceph user uid/gid from container image... 2024-04-14T13:27:24.132 INFO:teuthology.orchestra.run.smithi008.stderr:stat: 167 167 2024-04-14T13:27:24.580 INFO:teuthology.orchestra.run.smithi008.stderr:Creating initial keys... 2024-04-14T13:27:24.987 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph-authtool: AQC82RtmbkPGNRAAWreWdquWAvflRT9Yr/f5Wg== 2024-04-14T13:27:25.842 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph-authtool: AQC92RtmZX6gLBAAKuZ0h3qmxXEOSlf5yXg8lA== 2024-04-14T13:27:26.765 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph-authtool: AQC+2RtmSzCLKhAAmOfJkQ33kGoc7/vmlyZdJQ== 2024-04-14T13:27:27.305 INFO:teuthology.orchestra.run.smithi008.stderr:Creating initial monmap... 2024-04-14T13:27:27.679 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-14T13:27:27.679 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-14T13:27:27.679 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:27:27.679 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-14T13:27:28.143 INFO:teuthology.orchestra.run.smithi008.stderr:monmaptool for smithi008 [v2:172.21.15.8:3300,v1:172.21.15.8:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-14T13:27:28.143 INFO:teuthology.orchestra.run.smithi008.stderr:setting min_mon_release = octopus 2024-04-14T13:27:28.144 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/monmaptool: set fsid to a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:27:28.144 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-14T13:27:28.144 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-14T13:27:28.144 INFO:teuthology.orchestra.run.smithi008.stderr:Creating mon... 2024-04-14T13:27:29.015 INFO:teuthology.orchestra.run.smithi008.stderr:create mon.smithi008 on 2024-04-14T13:27:29.246 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-14T13:27:29.395 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968.target → /etc/systemd/system/ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968.target. 2024-04-14T13:27:29.395 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968.target → /etc/systemd/system/ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968.target. 2024-04-14T13:27:29.694 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Failed to reset failed state of unit ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@mon.smithi008.service: Unit ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@mon.smithi008.service not loaded. 2024-04-14T13:27:29.701 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Created symlink /etc/systemd/system/ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968.target.wants/ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@mon.smithi008.service → /etc/systemd/system/ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@.service. 2024-04-14T13:27:30.325 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:30 smithi008 ceph-mon[24441]: mon.smithi008 is new leader, mons smithi008 in quorum (ranks 0) 2024-04-14T13:27:30.337 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: disabled 2024-04-14T13:27:30.343 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: inactive 2024-04-14T13:27:30.344 INFO:teuthology.orchestra.run.smithi008.stderr:firewalld.service is not enabled 2024-04-14T13:27:30.344 INFO:teuthology.orchestra.run.smithi008.stderr:Not possible to enable service . firewalld.service is not available 2024-04-14T13:27:30.344 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for mon to start... 2024-04-14T13:27:30.344 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for mon... 2024-04-14T13:27:30.576 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:30 smithi008 bash[24423]: b66a8e3d787561a1ff5b29b256137c6320357aff883250d7e1bdd5f00bd50e5f 2024-04-14T13:27:30.576 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:30 smithi008 systemd[1]: Started Ceph mon.smithi008 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:27:30.839 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: cluster: 2024-04-14T13:27:30.839 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: id: a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:27:30.839 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-14T13:27:30.839 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-14T13:27:30.839 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: services: 2024-04-14T13:27:30.839 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi008 (age 0.573302s) 2024-04-14T13:27:30.839 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-14T13:27:30.839 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-14T13:27:30.839 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-14T13:27:30.839 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: data: 2024-04-14T13:27:30.839 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-14T13:27:30.839 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-14T13:27:30.839 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-14T13:27:30.840 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: pgs: 2024-04-14T13:27:30.840 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-14T13:27:31.250 INFO:teuthology.orchestra.run.smithi008.stderr:mon is available 2024-04-14T13:27:31.250 INFO:teuthology.orchestra.run.smithi008.stderr:Assimilating anything we can from ceph.conf... 2024-04-14T13:27:31.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:31 smithi008 ceph-mon[24441]: mon.smithi008 is new leader, mons smithi008 in quorum (ranks 0) 2024-04-14T13:27:31.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:31 smithi008 ceph-mon[24441]: monmap e1: 1 mons at {smithi008=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0]} 2024-04-14T13:27:31.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:31 smithi008 ceph-mon[24441]: fsmap 2024-04-14T13:27:31.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:31 smithi008 ceph-mon[24441]: osdmap e1: 0 total, 0 up, 0 in 2024-04-14T13:27:31.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:31 smithi008 ceph-mon[24441]: mgrmap e1: no daemons active 2024-04-14T13:27:31.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:31 smithi008 ceph-mon[24441]: from='client.? 172.21.15.8:0/2632961614' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-14T13:27:32.038 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-14T13:27:32.038 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: [global] 2024-04-14T13:27:32.038 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: fsid = a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:27:32.038 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.8:3300,v1:172.21.15.8:6789] 2024-04-14T13:27:32.038 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-14T13:27:32.038 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-14T13:27:32.038 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-14T13:27:32.038 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-14T13:27:32.038 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-14T13:27:32.038 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: [mgr] 2024-04-14T13:27:32.039 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-14T13:27:32.039 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-14T13:27:32.039 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: [osd] 2024-04-14T13:27:32.039 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-14T13:27:32.039 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-14T13:27:32.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:32 smithi008 ceph-mon[24441]: from='client.? 172.21.15.8:0/3482726017' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-14T13:27:32.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:32 smithi008 ceph-mon[24441]: from='client.? 172.21.15.8:0/3482726017' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-14T13:27:32.523 INFO:teuthology.orchestra.run.smithi008.stderr:Generating new minimal ceph.conf... 2024-04-14T13:27:33.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:33 smithi008 ceph-mon[24441]: from='client.? 172.21.15.8:0/928254271' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:27:33.497 INFO:teuthology.orchestra.run.smithi008.stderr:Restarting the monitor... 2024-04-14T13:27:33.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:33 smithi008 systemd[1]: Stopping Ceph mon.smithi008 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:27:33.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:33 smithi008 bash[24661]: Error: no container with name or ID "ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon.smithi008" found: no such container 2024-04-14T13:27:33.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:33 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008[24437]: 2024-04-14T13:27:33.654+0000 7f881dc53700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi008 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:27:33.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:33 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008[24437]: 2024-04-14T13:27:33.654+0000 7f881dc53700 -1 mon.smithi008@0(leader) e1 *** Got Signal Terminated *** 2024-04-14T13:27:33.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:33 smithi008 podman[24674]: 2024-04-14 13:27:33.7056572 +0000 UTC m=+0.151958616 container died b66a8e3d787561a1ff5b29b256137c6320357aff883250d7e1bdd5f00bd50e5f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, vcs-type=git, ceph=True, io.openshift.tags=base centos centos-stream, version=8, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., name=centos-stream) 2024-04-14T13:27:34.163 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:33 smithi008 podman[24674]: 2024-04-14 13:27:33.953556954 +0000 UTC m=+0.399858360 container cleanup b66a8e3d787561a1ff5b29b256137c6320357aff883250d7e1bdd5f00bd50e5f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, ceph=True, vcs-type=git, release=754, architecture=x86_64, name=centos-stream, GIT_CLEAN=True, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0) 2024-04-14T13:27:34.163 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:33 smithi008 bash[24674]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008 2024-04-14T13:27:34.163 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:33 smithi008 bash[24700]: Error: no container with name or ID "ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon.smithi008" found: no such container 2024-04-14T13:27:34.472 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:34 smithi008 podman[24688]: 2024-04-14 13:27:34.162620679 +0000 UTC m=+0.456819693 container remove b66a8e3d787561a1ff5b29b256137c6320357aff883250d7e1bdd5f00bd50e5f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, ceph=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, vendor=Red Hat, Inc., release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, io.buildah.version=1.19.8, io.openshift.expose-services=, architecture=x86_64, build-date=2022-05-03T08:36:31.336870) 2024-04-14T13:27:34.472 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:34 smithi008 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@mon.smithi008.service: Deactivated successfully. 2024-04-14T13:27:34.472 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:34 smithi008 systemd[1]: Stopped Ceph mon.smithi008 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:27:34.472 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:34 smithi008 systemd[1]: Starting Ceph mon.smithi008 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:27:34.473 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:34 smithi008 podman[24767]: 2024-04-14 13:27:34.371693366 +0000 UTC m=+0.087152611 container create 7786c27a60d0a0461d3eaa074797b473b59a422a1712ad98961c582e7de3898e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008, ceph=True, distribution-scope=public, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, version=8, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-14T13:27:34.473 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:34 smithi008 podman[24767]: 2024-04-14 13:27:34.303315433 +0000 UTC m=+0.018774671 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-14T13:27:35.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 podman[24767]: 2024-04-14 13:27:35.402343362 +0000 UTC m=+1.117802607 container init 7786c27a60d0a0461d3eaa074797b473b59a422a1712ad98961c582e7de3898e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008, release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, version=8, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, architecture=x86_64, distribution-scope=public, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-14T13:27:35.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 podman[24767]: 2024-04-14 13:27:35.406322673 +0000 UTC m=+1.121781910 container start 7786c27a60d0a0461d3eaa074797b473b59a422a1712ad98961c582e7de3898e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008, RELEASE=HEAD, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, version=8, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, name=centos-stream, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-14T13:27:35.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: set uid:gid to 167:167 (ceph:ceph) 2024-04-14T13:27:35.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-14T13:27:35.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: pidfile_write: ignore empty --pid-file 2024-04-14T13:27:35.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: load: jerasure load: lrc 2024-04-14T13:27:35.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: RocksDB version: 6.15.5 2024-04-14T13:27:35.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-14T13:27:35.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Compile date Apr 18 2022 2024-04-14T13:27:35.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: DB SUMMARY 2024-04-14T13:27:35.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: DB Session ID: LK8P9K2ULJLT3HADRO4X 2024-04-14T13:27:35.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: CURRENT file: CURRENT 2024-04-14T13:27:35.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: IDENTITY file: IDENTITY 2024-04-14T13:27:35.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-14T13:27:35.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi008/store.db dir, Total Num: 1, files: 000008.sst 2024-04-14T13:27:35.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi008/store.db: 000010.log size: 82683 ; 2024-04-14T13:27:35.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.error_if_exists: 0 2024-04-14T13:27:35.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.create_if_missing: 0 2024-04-14T13:27:35.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.paranoid_checks: 1 2024-04-14T13:27:35.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-14T13:27:35.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.env: 0x55cbf6ac6860 2024-04-14T13:27:35.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.fs: Posix File System 2024-04-14T13:27:35.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.info_log: 0x55cbf8e21fa0 2024-04-14T13:27:35.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_file_opening_threads: 16 2024-04-14T13:27:35.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.statistics: (nil) 2024-04-14T13:27:35.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.use_fsync: 0 2024-04-14T13:27:35.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_log_file_size: 0 2024-04-14T13:27:35.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-14T13:27:35.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-14T13:27:35.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.keep_log_file_num: 1000 2024-04-14T13:27:35.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.recycle_log_file_num: 0 2024-04-14T13:27:35.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.allow_fallocate: 1 2024-04-14T13:27:35.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.allow_mmap_reads: 0 2024-04-14T13:27:35.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.allow_mmap_writes: 0 2024-04-14T13:27:35.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.use_direct_reads: 0 2024-04-14T13:27:35.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-14T13:27:35.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.create_missing_column_families: 0 2024-04-14T13:27:35.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.db_log_dir: 2024-04-14T13:27:35.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi008/store.db 2024-04-14T13:27:35.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-14T13:27:35.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-14T13:27:35.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-14T13:27:35.855 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-14T13:27:35.855 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-14T13:27:35.855 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-14T13:27:35.855 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.advise_random_on_open: 1 2024-04-14T13:27:35.856 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.db_write_buffer_size: 0 2024-04-14T13:27:35.856 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.write_buffer_manager: 0x55cbf8f12330 2024-04-14T13:27:35.856 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-14T13:27:35.856 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-14T13:27:35.856 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-14T13:27:35.856 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-14T13:27:35.856 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.rate_limiter: (nil) 2024-04-14T13:27:35.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-14T13:27:35.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.wal_recovery_mode: 2 2024-04-14T13:27:35.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.enable_thread_tracking: 0 2024-04-14T13:27:35.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.enable_pipelined_write: 0 2024-04-14T13:27:35.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.unordered_write: 0 2024-04-14T13:27:35.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-14T13:27:35.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-14T13:27:35.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-14T13:27:35.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-14T13:27:35.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.row_cache: None 2024-04-14T13:27:35.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.wal_filter: None 2024-04-14T13:27:35.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-14T13:27:35.858 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.allow_ingest_behind: 0 2024-04-14T13:27:35.858 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.preserve_deletes: 0 2024-04-14T13:27:35.858 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.two_write_queues: 0 2024-04-14T13:27:35.858 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.manual_wal_flush: 0 2024-04-14T13:27:35.858 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.atomic_flush: 0 2024-04-14T13:27:35.858 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-14T13:27:35.858 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-14T13:27:35.858 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-14T13:27:35.858 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.log_readahead_size: 0 2024-04-14T13:27:35.858 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-14T13:27:35.858 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.best_efforts_recovery: 0 2024-04-14T13:27:35.858 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-14T13:27:35.858 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-14T13:27:35.858 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.allow_data_in_errors: 0 2024-04-14T13:27:35.858 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.db_host_id: __hostname__ 2024-04-14T13:27:35.859 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_background_jobs: 2 2024-04-14T13:27:35.859 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_background_compactions: -1 2024-04-14T13:27:35.859 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_subcompactions: 1 2024-04-14T13:27:35.859 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-14T13:27:35.859 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-14T13:27:35.859 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-14T13:27:35.859 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_total_wal_size: 0 2024-04-14T13:27:35.859 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-14T13:27:35.859 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-14T13:27:35.859 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-14T13:27:35.859 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-14T13:27:35.859 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_open_files: -1 2024-04-14T13:27:35.859 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.bytes_per_sync: 0 2024-04-14T13:27:35.859 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-14T13:27:35.859 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-14T13:27:35.859 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compaction_readahead_size: 0 2024-04-14T13:27:35.860 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_background_flushes: -1 2024-04-14T13:27:35.860 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Compression algorithms supported: 2024-04-14T13:27:35.860 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-14T13:27:35.860 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: kZSTD supported: 0 2024-04-14T13:27:35.860 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: kXpressCompression supported: 0 2024-04-14T13:27:35.860 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: kLZ4HCCompression supported: 1 2024-04-14T13:27:35.860 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: kLZ4Compression supported: 1 2024-04-14T13:27:35.860 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: kBZip2Compression supported: 0 2024-04-14T13:27:35.860 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: kZlibCompression supported: 1 2024-04-14T13:27:35.860 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: kSnappyCompression supported: 1 2024-04-14T13:27:35.860 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-14T13:27:35.860 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi008/store.db/MANIFEST-000009 2024-04-14T13:27:35.860 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-14T13:27:35.860 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-14T13:27:35.860 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.merge_operator: 2024-04-14T13:27:35.861 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compaction_filter: None 2024-04-14T13:27:35.861 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compaction_filter_factory: None 2024-04-14T13:27:35.861 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.sst_partitioner_factory: None 2024-04-14T13:27:35.861 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-14T13:27:35.861 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-14T13:27:35.861 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cbf8dedd00) 2024-04-14T13:27:35.861 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: cache_index_and_filter_blocks: 1 2024-04-14T13:27:35.861 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-14T13:27:35.861 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-14T13:27:35.861 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: pin_top_level_index_and_filter: 1 2024-04-14T13:27:35.861 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: index_type: 0 2024-04-14T13:27:35.861 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: data_block_index_type: 0 2024-04-14T13:27:35.861 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: index_shortening: 1 2024-04-14T13:27:35.861 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-14T13:27:35.861 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: hash_index_allow_collision: 1 2024-04-14T13:27:35.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: checksum: 1 2024-04-14T13:27:35.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: no_block_cache: 0 2024-04-14T13:27:35.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache: 0x55cbf8e58170 2024-04-14T13:27:35.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache_name: BinnedLRUCache 2024-04-14T13:27:35.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache_options: 2024-04-14T13:27:35.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: capacity : 536870912 2024-04-14T13:27:35.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: num_shard_bits : 4 2024-04-14T13:27:35.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: strict_capacity_limit : 0 2024-04-14T13:27:35.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: high_pri_pool_ratio: 0.000 2024-04-14T13:27:35.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache_compressed: (nil) 2024-04-14T13:27:35.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: persistent_cache: (nil) 2024-04-14T13:27:35.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_size: 4096 2024-04-14T13:27:35.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_size_deviation: 10 2024-04-14T13:27:35.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_restart_interval: 16 2024-04-14T13:27:35.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: index_block_restart_interval: 1 2024-04-14T13:27:35.863 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: metadata_block_size: 4096 2024-04-14T13:27:35.863 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: partition_filters: 0 2024-04-14T13:27:35.863 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: use_delta_encoding: 1 2024-04-14T13:27:35.863 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-14T13:27:35.863 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: whole_key_filtering: 1 2024-04-14T13:27:35.863 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: verify_compression: 0 2024-04-14T13:27:35.863 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: read_amp_bytes_per_bit: 0 2024-04-14T13:27:35.863 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: format_version: 4 2024-04-14T13:27:35.863 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: enable_index_compression: 1 2024-04-14T13:27:35.863 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_align: 0 2024-04-14T13:27:35.863 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.write_buffer_size: 33554432 2024-04-14T13:27:35.863 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_write_buffer_number: 2 2024-04-14T13:27:35.863 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compression: NoCompression 2024-04-14T13:27:35.863 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.bottommost_compression: Disabled 2024-04-14T13:27:35.863 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.prefix_extractor: nullptr 2024-04-14T13:27:35.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-14T13:27:35.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.num_levels: 7 2024-04-14T13:27:35.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-14T13:27:35.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-14T13:27:35.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-14T13:27:35.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-14T13:27:35.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-14T13:27:35.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-14T13:27:35.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-14T13:27:35.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-14T13:27:35.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-14T13:27:35.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-14T13:27:35.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-14T13:27:35.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compression_opts.level: 32767 2024-04-14T13:27:35.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compression_opts.strategy: 0 2024-04-14T13:27:35.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-14T13:27:35.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-14T13:27:35.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-14T13:27:35.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compression_opts.enabled: false 2024-04-14T13:27:35.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-14T13:27:35.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-14T13:27:35.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-14T13:27:35.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.target_file_size_base: 67108864 2024-04-14T13:27:35.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-14T13:27:35.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-14T13:27:35.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-14T13:27:35.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-14T13:27:35.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-14T13:27:35.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-14T13:27:35.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-14T13:27:35.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-14T13:27:35.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-14T13:27:35.866 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-14T13:27:35.866 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-14T13:27:35.866 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-14T13:27:35.866 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-14T13:27:35.866 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.arena_block_size: 4194304 2024-04-14T13:27:35.866 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-14T13:27:35.866 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-14T13:27:35.866 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-14T13:27:35.866 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.disable_auto_compactions: 0 2024-04-14T13:27:35.866 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-14T13:27:35.866 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-14T13:27:35.866 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-14T13:27:35.866 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-14T13:27:35.866 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-14T13:27:35.866 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-14T13:27:35.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-14T13:27:35.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-14T13:27:35.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-14T13:27:35.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-14T13:27:35.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.table_properties_collectors: 2024-04-14T13:27:35.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.inplace_update_support: 0 2024-04-14T13:27:35.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-14T13:27:35.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-14T13:27:35.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-14T13:27:35.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-14T13:27:35.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.bloom_locality: 0 2024-04-14T13:27:35.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.max_successive_merges: 0 2024-04-14T13:27:35.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-14T13:27:35.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.paranoid_file_checks: 0 2024-04-14T13:27:35.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.force_consistency_checks: 1 2024-04-14T13:27:35.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.report_bg_io_stats: 0 2024-04-14T13:27:35.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.ttl: 2592000 2024-04-14T13:27:35.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-14T13:27:35.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.enable_blob_files: false 2024-04-14T13:27:35.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.min_blob_size: 0 2024-04-14T13:27:35.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.blob_file_size: 268435456 2024-04-14T13:27:35.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-14T13:27:35.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-14T13:27:35.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-14T13:27:35.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi008/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-14T13:27:35.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-14T13:27:35.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-14T13:27:35.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713101255436890, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-14T13:27:35.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-14T13:27:35.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-14T13:27:35.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713101255437724, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79365, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77629, "index_size": 224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9884, "raw_average_key_size": 47, "raw_value_size": 71938, "raw_average_value_size": 344, "num_data_blocks": 10, "num_entries": 209, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713101255, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "8baabb18-a9dc-4b86-9195-2a945687bc19", "db_session_id": "LK8P9K2ULJLT3HADRO4X"}} 2024-04-14T13:27:35.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-14T13:27:35.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713101255438172, "job": 1, "event": "recovery_finished"} 2024-04-14T13:27:35.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi008/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-14T13:27:35.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55cbf8e3ea80 2024-04-14T13:27:35.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: DB pointer 0x55cbf8e4e000 2024-04-14T13:27:35.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-14T13:27:35.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-14T13:27:35.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** DB Stats ** 2024-04-14T13:27:35.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T13:27:35.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-14T13:27:35.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T13:27:35.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T13:27:35.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-14T13:27:35.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-14T13:27:35.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T13:27:35.870 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-14T13:27:35.870 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** Compaction Stats [default] ** 2024-04-14T13:27:35.870 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T13:27:35.870 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T13:27:35.870 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: L0 2/0 79.30 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.0 0.00 0.00 1 0.001 0 0 2024-04-14T13:27:35.870 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Sum 2/0 79.30 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.0 0.00 0.00 1 0.001 0 0 2024-04-14T13:27:35.870 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.0 0.00 0.00 1 0.001 0 0 2024-04-14T13:27:35.870 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-14T13:27:35.870 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** Compaction Stats [default] ** 2024-04-14T13:27:35.870 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T13:27:35.870 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T13:27:35.870 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 133.0 0.00 0.00 1 0.001 0 0 2024-04-14T13:27:35.870 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T13:27:35.870 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-14T13:27:35.871 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T13:27:35.871 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T13:27:35.871 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T13:27:35.871 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T13:27:35.871 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative compaction: 0.00 GB write, 20.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T13:27:35.871 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval compaction: 0.00 GB write, 20.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T13:27:35.871 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T13:27:35.871 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-14T13:27:35.871 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T13:27:35.871 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-14T13:27:35.871 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** Compaction Stats [default] ** 2024-04-14T13:27:35.871 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T13:27:35.871 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T13:27:35.871 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: L0 2/0 79.30 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.0 0.00 0.00 1 0.001 0 0 2024-04-14T13:27:35.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Sum 2/0 79.30 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.0 0.00 0.00 1 0.001 0 0 2024-04-14T13:27:35.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-14T13:27:35.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-14T13:27:35.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** Compaction Stats [default] ** 2024-04-14T13:27:35.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T13:27:35.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T13:27:35.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 133.0 0.00 0.00 1 0.001 0 0 2024-04-14T13:27:35.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T13:27:35.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-14T13:27:35.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T13:27:35.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T13:27:35.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T13:27:35.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T13:27:35.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative compaction: 0.00 GB write, 20.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T13:27:35.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T13:27:35.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T13:27:35.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-14T13:27:35.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T13:27:35.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: starting mon.smithi008 rank 0 at public addrs [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] at bind addrs [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi008 fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:27:35.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: mon.smithi008@-1(???) e1 preinit fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:27:35.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: mon.smithi008@-1(???).mds e1 new map 2024-04-14T13:27:35.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: mon.smithi008@-1(???).mds e1 print_map 2024-04-14T13:27:35.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: e1 2024-04-14T13:27:35.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-14T13:27:35.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-14T13:27:35.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: legacy client fscid: -1 2024-04-14T13:27:35.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-14T13:27:35.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: No filesystems configured 2024-04-14T13:27:35.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: mon.smithi008@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-14T13:27:35.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: mon.smithi008@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T13:27:35.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: mon.smithi008@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T13:27:35.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: mon.smithi008@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T13:27:35.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: mon.smithi008@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-14T13:27:35.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: expand_channel_meta expand map: {default=false} 2024-04-14T13:27:35.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: expand_channel_meta from 'false' to 'false' 2024-04-14T13:27:35.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: expand_channel_meta expanded map: {default=false} 2024-04-14T13:27:35.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: expand_channel_meta expand map: {default=info} 2024-04-14T13:27:35.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: expand_channel_meta from 'info' to 'info' 2024-04-14T13:27:35.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: expand_channel_meta expanded map: {default=info} 2024-04-14T13:27:35.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: expand_channel_meta expand map: {default=daemon} 2024-04-14T13:27:35.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-14T13:27:35.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: expand_channel_meta expanded map: {default=daemon} 2024-04-14T13:27:35.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: expand_channel_meta expand map: {default=debug} 2024-04-14T13:27:35.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: expand_channel_meta from 'debug' to 'debug' 2024-04-14T13:27:35.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: expand_channel_meta expanded map: {default=debug} 2024-04-14T13:27:35.875 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: mon.smithi008 is new leader, mons smithi008 in quorum (ranks 0) 2024-04-14T13:27:35.875 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: monmap e1: 1 mons at {smithi008=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0]} 2024-04-14T13:27:35.875 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: fsmap 2024-04-14T13:27:35.875 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: osdmap e1: 0 total, 0 up, 0 in 2024-04-14T13:27:35.875 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:35 smithi008 ceph-mon[24786]: mgrmap e1: no daemons active 2024-04-14T13:27:36.248 INFO:teuthology.orchestra.run.smithi008.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-14T13:27:36.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:36 smithi008 bash[24767]: 7786c27a60d0a0461d3eaa074797b473b59a422a1712ad98961c582e7de3898e 2024-04-14T13:27:36.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:36 smithi008 systemd[1]: Started Ceph mon.smithi008 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:27:40.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:40 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2151847562' entity='client.admin' 2024-04-14T13:27:45.833 INFO:teuthology.orchestra.run.smithi008.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-14T13:27:45.833 INFO:teuthology.orchestra.run.smithi008.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-14T13:27:45.833 INFO:teuthology.orchestra.run.smithi008.stderr:Creating mgr... 2024-04-14T13:27:45.833 INFO:teuthology.orchestra.run.smithi008.stderr:Verifying port 9283 ... 2024-04-14T13:27:46.123 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Failed to reset failed state of unit ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@mgr.smithi008.yemlhr.service: Unit ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@mgr.smithi008.yemlhr.service not loaded. 2024-04-14T13:27:46.130 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Created symlink /etc/systemd/system/ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968.target.wants/ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@mgr.smithi008.yemlhr.service → /etc/systemd/system/ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@.service. 2024-04-14T13:27:46.931 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: disabled 2024-04-14T13:27:46.938 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: inactive 2024-04-14T13:27:46.938 INFO:teuthology.orchestra.run.smithi008.stderr:firewalld.service is not enabled 2024-04-14T13:27:46.938 INFO:teuthology.orchestra.run.smithi008.stderr:Not possible to enable service . firewalld.service is not available 2024-04-14T13:27:46.946 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: disabled 2024-04-14T13:27:46.953 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: inactive 2024-04-14T13:27:46.953 INFO:teuthology.orchestra.run.smithi008.stderr:firewalld.service is not enabled 2024-04-14T13:27:46.954 INFO:teuthology.orchestra.run.smithi008.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-14T13:27:46.954 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for mgr to start... 2024-04-14T13:27:46.954 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for mgr... 2024-04-14T13:27:47.515 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-14T13:27:47.516 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-04-14T13:27:47.516 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsid": "a8b03f6a-fa62-11ee-bc8e-c7b262605968", 2024-04-14T13:27:47.516 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "health": { 2024-04-14T13:27:47.516 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-14T13:27:47.516 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "checks": {}, 2024-04-14T13:27:47.516 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mutes": [] 2024-04-14T13:27:47.516 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:47.516 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-14T13:27:47.516 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum": [ 2024-04-14T13:27:47.516 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 0 2024-04-14T13:27:47.517 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-14T13:27:47.517 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-14T13:27:47.517 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "smithi008" 2024-04-14T13:27:47.517 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-14T13:27:47.517 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_age": 12, 2024-04-14T13:27:47.517 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "monmap": { 2024-04-14T13:27:47.517 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:27:47.517 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-14T13:27:47.517 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-14T13:27:47.517 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:47.517 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osdmap": { 2024-04-14T13:27:47.518 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:27:47.518 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-14T13:27:47.518 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-14T13:27:47.518 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-14T13:27:47.518 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-14T13:27:47.518 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-14T13:27:47.518 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-14T13:27:47.518 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:47.518 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgmap": { 2024-04-14T13:27:47.518 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-14T13:27:47.518 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-14T13:27:47.519 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-14T13:27:47.519 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-14T13:27:47.519 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-14T13:27:47.519 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-14T13:27:47.519 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-14T13:27:47.519 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-14T13:27:47.519 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:47.519 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsmap": { 2024-04-14T13:27:47.519 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:27:47.519 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-14T13:27:47.519 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-14T13:27:47.519 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:47.520 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-14T13:27:47.520 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "available": false, 2024-04-14T13:27:47.520 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-14T13:27:47.520 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modules": [ 2024-04-14T13:27:47.520 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "iostat", 2024-04-14T13:27:47.520 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "nfs", 2024-04-14T13:27:47.520 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "restful" 2024-04-14T13:27:47.520 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-14T13:27:47.520 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-04-14T13:27:47.520 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:47.520 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "servicemap": { 2024-04-14T13:27:47.520 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:27:47.521 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modified": "2024-04-14T13:27:30.263487+0000", 2024-04-14T13:27:47.521 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-04-14T13:27:47.521 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:47.521 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-14T13:27:47.521 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-04-14T13:27:47.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:47 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/648653565' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-14T13:27:48.084 INFO:teuthology.orchestra.run.smithi008.stderr:mgr not available, waiting (1/15)... 2024-04-14T13:27:50.608 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-14T13:27:50.609 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-04-14T13:27:50.609 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsid": "a8b03f6a-fa62-11ee-bc8e-c7b262605968", 2024-04-14T13:27:50.609 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "health": { 2024-04-14T13:27:50.609 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-14T13:27:50.609 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "checks": {}, 2024-04-14T13:27:50.609 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mutes": [] 2024-04-14T13:27:50.609 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:50.609 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-14T13:27:50.609 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum": [ 2024-04-14T13:27:50.609 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 0 2024-04-14T13:27:50.609 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-14T13:27:50.609 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-14T13:27:50.609 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "smithi008" 2024-04-14T13:27:50.610 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-14T13:27:50.610 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_age": 15, 2024-04-14T13:27:50.610 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "monmap": { 2024-04-14T13:27:50.610 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:27:50.610 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-14T13:27:50.610 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-14T13:27:50.610 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:50.610 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osdmap": { 2024-04-14T13:27:50.610 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:27:50.610 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-14T13:27:50.610 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-14T13:27:50.610 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-14T13:27:50.610 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-14T13:27:50.614 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-14T13:27:50.614 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-14T13:27:50.614 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:50.614 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgmap": { 2024-04-14T13:27:50.614 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-14T13:27:50.614 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-14T13:27:50.614 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-14T13:27:50.614 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-14T13:27:50.614 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-14T13:27:50.614 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-14T13:27:50.614 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-14T13:27:50.615 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-14T13:27:50.615 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:50.615 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsmap": { 2024-04-14T13:27:50.615 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:27:50.615 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-14T13:27:50.615 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-14T13:27:50.615 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:50.615 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-14T13:27:50.615 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "available": false, 2024-04-14T13:27:50.615 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-14T13:27:50.615 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modules": [ 2024-04-14T13:27:50.615 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "iostat", 2024-04-14T13:27:50.615 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "nfs", 2024-04-14T13:27:50.615 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "restful" 2024-04-14T13:27:50.615 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-14T13:27:50.615 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-04-14T13:27:50.616 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:50.616 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "servicemap": { 2024-04-14T13:27:50.616 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:27:50.616 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modified": "2024-04-14T13:27:30.263487+0000", 2024-04-14T13:27:50.616 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-04-14T13:27:50.616 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:50.616 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-14T13:27:50.616 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-04-14T13:27:51.075 INFO:teuthology.orchestra.run.smithi008.stderr:mgr not available, waiting (2/15)... 2024-04-14T13:27:51.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:51 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/3241704389' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-14T13:27:53.200 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:52 smithi008 ceph-mon[24786]: Activating manager daemon smithi008.yemlhr 2024-04-14T13:27:53.200 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:52 smithi008 ceph-mon[24786]: mgrmap e2: smithi008.yemlhr(active, starting, since 0.00269605s) 2024-04-14T13:27:53.200 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:52 smithi008 ceph-mon[24786]: from='mgr.14100 172.21.15.8:0/1416172011' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T13:27:53.200 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:52 smithi008 ceph-mon[24786]: from='mgr.14100 172.21.15.8:0/1416172011' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T13:27:53.200 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:52 smithi008 ceph-mon[24786]: from='mgr.14100 172.21.15.8:0/1416172011' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T13:27:53.200 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:52 smithi008 ceph-mon[24786]: from='mgr.14100 172.21.15.8:0/1416172011' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:27:53.200 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:52 smithi008 ceph-mon[24786]: from='mgr.14100 172.21.15.8:0/1416172011' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr metadata", "who": "smithi008.yemlhr", "id": "smithi008.yemlhr"}]: dispatch 2024-04-14T13:27:53.200 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:52 smithi008 ceph-mon[24786]: Manager daemon smithi008.yemlhr is now available 2024-04-14T13:27:53.200 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:52 smithi008 ceph-mon[24786]: from='mgr.14100 172.21.15.8:0/1416172011' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:27:53.200 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:52 smithi008 ceph-mon[24786]: from='mgr.14100 172.21.15.8:0/1416172011' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:27:53.201 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:52 smithi008 ceph-mon[24786]: from='mgr.14100 172.21.15.8:0/1416172011' entity='mgr.smithi008.yemlhr' 2024-04-14T13:27:53.201 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:52 smithi008 ceph-mon[24786]: from='mgr.14100 172.21.15.8:0/1416172011' entity='mgr.smithi008.yemlhr' 2024-04-14T13:27:53.201 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:52 smithi008 ceph-mon[24786]: from='mgr.14100 172.21.15.8:0/1416172011' entity='mgr.smithi008.yemlhr' 2024-04-14T13:27:53.579 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-14T13:27:53.579 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-04-14T13:27:53.579 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsid": "a8b03f6a-fa62-11ee-bc8e-c7b262605968", 2024-04-14T13:27:53.579 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "health": { 2024-04-14T13:27:53.579 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-14T13:27:53.579 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "checks": {}, 2024-04-14T13:27:53.579 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mutes": [] 2024-04-14T13:27:53.579 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:53.579 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-14T13:27:53.579 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum": [ 2024-04-14T13:27:53.580 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 0 2024-04-14T13:27:53.580 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-14T13:27:53.580 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-14T13:27:53.580 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "smithi008" 2024-04-14T13:27:53.580 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-14T13:27:53.580 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_age": 18, 2024-04-14T13:27:53.580 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "monmap": { 2024-04-14T13:27:53.580 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:27:53.580 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-14T13:27:53.580 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-14T13:27:53.580 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:53.580 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osdmap": { 2024-04-14T13:27:53.580 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:27:53.580 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-14T13:27:53.580 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-14T13:27:53.581 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-14T13:27:53.584 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-14T13:27:53.584 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-14T13:27:53.584 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-14T13:27:53.584 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:53.584 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgmap": { 2024-04-14T13:27:53.584 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-14T13:27:53.585 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-14T13:27:53.585 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-14T13:27:53.585 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-14T13:27:53.585 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-14T13:27:53.585 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-14T13:27:53.585 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-14T13:27:53.585 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-14T13:27:53.585 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:53.585 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsmap": { 2024-04-14T13:27:53.585 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:27:53.585 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-14T13:27:53.585 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-14T13:27:53.585 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:53.585 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-14T13:27:53.585 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "available": false, 2024-04-14T13:27:53.586 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-14T13:27:53.586 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modules": [ 2024-04-14T13:27:53.586 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "iostat", 2024-04-14T13:27:53.586 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "nfs", 2024-04-14T13:27:53.586 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "restful" 2024-04-14T13:27:53.586 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-14T13:27:53.586 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-04-14T13:27:53.586 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:53.586 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "servicemap": { 2024-04-14T13:27:53.586 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:27:53.586 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modified": "2024-04-14T13:27:30.263487+0000", 2024-04-14T13:27:53.586 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-04-14T13:27:53.586 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:53.586 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-14T13:27:53.586 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-04-14T13:27:54.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:53 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2942620770' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-14T13:27:54.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:53 smithi008 ceph-mon[24786]: mgrmap e3: smithi008.yemlhr(active, since 1.00587s) 2024-04-14T13:27:54.185 INFO:teuthology.orchestra.run.smithi008.stderr:mgr not available, waiting (3/15)... 2024-04-14T13:27:56.301 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:55 smithi008 ceph-mon[24786]: mgrmap e4: smithi008.yemlhr(active, since 2s) 2024-04-14T13:27:56.856 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-14T13:27:56.857 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-04-14T13:27:56.857 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsid": "a8b03f6a-fa62-11ee-bc8e-c7b262605968", 2024-04-14T13:27:56.857 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "health": { 2024-04-14T13:27:56.857 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-14T13:27:56.857 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "checks": {}, 2024-04-14T13:27:56.857 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mutes": [] 2024-04-14T13:27:56.857 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:56.857 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-14T13:27:56.857 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum": [ 2024-04-14T13:27:56.857 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 0 2024-04-14T13:27:56.857 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-14T13:27:56.857 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-14T13:27:56.858 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "smithi008" 2024-04-14T13:27:56.858 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-14T13:27:56.858 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_age": 21, 2024-04-14T13:27:56.858 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "monmap": { 2024-04-14T13:27:56.858 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:27:56.858 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-14T13:27:56.858 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-14T13:27:56.858 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:56.858 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osdmap": { 2024-04-14T13:27:56.858 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:27:56.858 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-14T13:27:56.858 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-14T13:27:56.858 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-14T13:27:56.858 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-14T13:27:56.859 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-14T13:27:56.859 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-14T13:27:56.862 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:56.862 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgmap": { 2024-04-14T13:27:56.862 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-14T13:27:56.862 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-14T13:27:56.862 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-14T13:27:56.862 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-14T13:27:56.862 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-14T13:27:56.862 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-14T13:27:56.862 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-14T13:27:56.863 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-14T13:27:56.863 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:56.863 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsmap": { 2024-04-14T13:27:56.863 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:27:56.863 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-14T13:27:56.863 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-14T13:27:56.863 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:56.863 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-14T13:27:56.863 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "available": true, 2024-04-14T13:27:56.863 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-14T13:27:56.863 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modules": [ 2024-04-14T13:27:56.863 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "iostat", 2024-04-14T13:27:56.863 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "nfs", 2024-04-14T13:27:56.863 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "restful" 2024-04-14T13:27:56.864 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-14T13:27:56.864 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-04-14T13:27:56.864 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:56.864 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "servicemap": { 2024-04-14T13:27:56.864 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:27:56.864 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modified": "2024-04-14T13:27:30.263487+0000", 2024-04-14T13:27:56.864 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-04-14T13:27:56.864 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-14T13:27:56.864 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-14T13:27:56.864 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-04-14T13:27:57.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:56 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/245286914' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-14T13:27:57.650 INFO:teuthology.orchestra.run.smithi008.stderr:mgr is available 2024-04-14T13:27:58.248 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-14T13:27:58.248 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: [global] 2024-04-14T13:27:58.248 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: fsid = a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:27:58.248 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-14T13:27:58.248 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-14T13:27:58.248 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-14T13:27:58.248 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-14T13:27:58.248 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-14T13:27:58.248 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: [mgr] 2024-04-14T13:27:58.248 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-14T13:27:58.248 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-14T13:27:58.249 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: [osd] 2024-04-14T13:27:58.249 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-14T13:27:58.249 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-14T13:27:58.569 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:58 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/3432020178' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-14T13:27:59.157 INFO:teuthology.orchestra.run.smithi008.stderr:Enabling cephadm module... 2024-04-14T13:28:00.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:27:59 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2689073442' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-14T13:28:01.014 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:00 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2689073442' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-14T13:28:01.014 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:00 smithi008 ceph-mon[24786]: mgrmap e5: smithi008.yemlhr(active, since 7s) 2024-04-14T13:28:01.015 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-04-14T13:28:01.015 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-14T13:28:01.016 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "available": true, 2024-04-14T13:28:01.016 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "active_name": "smithi008.yemlhr", 2024-04-14T13:28:01.016 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-14T13:28:01.016 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-04-14T13:28:01.602 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for the mgr to restart... 2024-04-14T13:28:01.602 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for mgr epoch 5... 2024-04-14T13:28:02.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:01 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2140702449' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-14T13:28:07.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:06 smithi008 ceph-mon[24786]: Active manager daemon smithi008.yemlhr restarted 2024-04-14T13:28:07.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:06 smithi008 ceph-mon[24786]: Activating manager daemon smithi008.yemlhr 2024-04-14T13:28:07.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:06 smithi008 ceph-mon[24786]: osdmap e2: 0 total, 0 up, 0 in 2024-04-14T13:28:07.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:06 smithi008 ceph-mon[24786]: mgrmap e6: smithi008.yemlhr(active, starting, since 0.627553s) 2024-04-14T13:28:07.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:06 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:28:07.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:06 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr metadata", "who": "smithi008.yemlhr", "id": "smithi008.yemlhr"}]: dispatch 2024-04-14T13:28:07.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:06 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T13:28:07.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:06 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T13:28:07.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:06 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T13:28:07.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:06 smithi008 ceph-mon[24786]: Manager daemon smithi008.yemlhr is now available 2024-04-14T13:28:07.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:06 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:07.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:06 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:07.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:06 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:28:07.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:06 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:28:07.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:06 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:28:07.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:06 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:28:07.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:06 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:28:07.465 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-04-14T13:28:07.465 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-14T13:28:07.465 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "initialized": true 2024-04-14T13:28:07.465 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-04-14T13:28:08.009 INFO:teuthology.orchestra.run.smithi008.stderr:mgr epoch 5 is available 2024-04-14T13:28:08.009 INFO:teuthology.orchestra.run.smithi008.stderr:Setting orchestrator backend to cephadm... 2024-04-14T13:28:08.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:08 smithi008 ceph-mon[24786]: mgrmap e7: smithi008.yemlhr(active, since 1.62982s) 2024-04-14T13:28:08.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:08 smithi008 ceph-mon[24786]: [14/Apr/2024:13:28:07] ENGINE Bus STARTING 2024-04-14T13:28:08.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:08 smithi008 ceph-mon[24786]: [14/Apr/2024:13:28:07] ENGINE Serving on https://172.21.15.8:7150 2024-04-14T13:28:08.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:08 smithi008 ceph-mon[24786]: [14/Apr/2024:13:28:07] ENGINE Bus STARTED 2024-04-14T13:28:08.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:08 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:08.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:08 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:28:09.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:09 smithi008 ceph-mon[24786]: mgrmap e8: smithi008.yemlhr(active, since 2s) 2024-04-14T13:28:09.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:09 smithi008 ceph-mon[24786]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:28:09.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:09 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:09.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:09 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:28:09.863 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: value unchanged 2024-04-14T13:28:10.463 INFO:teuthology.orchestra.run.smithi008.stderr:Generating ssh key... 2024-04-14T13:28:10.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:10 smithi008 ceph-mon[24786]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:28:12.404 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:12 smithi008 ceph-mon[24786]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:28:12.405 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:12 smithi008 ceph-mon[24786]: Generating ssh key... 2024-04-14T13:28:12.405 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:12 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:12.405 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:12 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:12.405 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/pU9P0fE4tzIbdwggbfTUC2texBubxDHJELjKn1Th2+S/W17ZqXwNMpSj0AvFkmXN/EN6ouVcbPGCZ6wvxTGiV9qHA8dPCqlY5ebJhJmijHUsGyLX3lOXZqwTrA/8MFg2jA9clYgzzCHYe1Vqh1wbts6/BT/KDZ+eDk0r3fB9EWyLwZL9dRN1V4TkI8a+v2dUISZniiJK0g6FUODFsCclowARL75HcaZ29MRl4dkXni8/tRv8dZzuZ3x9OZjwb4pJ8tg8z4I0q4uZGIGw1IF9c8zTyQPVkYzn26DJ5GYAwL6Z+BpRA476EAarVLah8IamWoZdHmNN3/tWLrnzw/9b0Wy5X/dlMCtB/DuZdCSeD3DzaYE06Fdis5NvACpCSS+T1FRrn63fwi8angHzvBLeWzAratVJdn+ISySkDgW3PH0dXJ/9KMwJjIY27s3w2V0QnoFvJd03fiJa0c97/hENn0cg5+CYa7QIawZPKHT4tXM7F+m624bEp+Su3P7/SFU= ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:28:12.993 INFO:teuthology.orchestra.run.smithi008.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-14T13:28:12.993 INFO:teuthology.orchestra.run.smithi008.stderr:Adding key to root@localhost authorized_keys... 2024-04-14T13:28:12.994 INFO:teuthology.orchestra.run.smithi008.stderr:Adding host smithi008... 2024-04-14T13:28:13.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:13 smithi008 ceph-mon[24786]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:28:14.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:14 smithi008 ceph-mon[24786]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi008", "addr": "172.21.15.8", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:28:14.627 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Added host 'smithi008' with addr '172.21.15.8' 2024-04-14T13:28:15.207 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying mon service with default placement... 2024-04-14T13:28:15.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:15 smithi008 ceph-mon[24786]: Deploying cephadm binary to smithi008 2024-04-14T13:28:15.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:15 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:15.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:15 smithi008 ceph-mon[24786]: Added host smithi008 2024-04-14T13:28:15.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:15 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:28:15.802 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-14T13:28:16.609 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying mgr service with default placement... 2024-04-14T13:28:16.909 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:16 smithi008 ceph-mon[24786]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:28:16.909 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:16 smithi008 ceph-mon[24786]: Saving service mon spec with placement count:5 2024-04-14T13:28:16.909 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:16 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:17.218 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-14T13:28:18.403 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:18 smithi008 ceph-mon[24786]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:28:18.403 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:18 smithi008 ceph-mon[24786]: Saving service mgr spec with placement count:2 2024-04-14T13:28:18.404 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:18 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:18.579 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying crash service with default placement... 2024-04-14T13:28:19.226 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled crash update... 2024-04-14T13:28:19.847 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:19 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:19.847 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:19 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:19.848 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:19 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:20.030 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying prometheus service with default placement... 2024-04-14T13:28:20.675 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:20 smithi008 ceph-mon[24786]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:28:20.675 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:20 smithi008 ceph-mon[24786]: Saving service crash spec with placement * 2024-04-14T13:28:20.886 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-04-14T13:28:21.479 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying grafana service with default placement... 2024-04-14T13:28:21.747 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:21 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:21.747 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:21 smithi008 ceph-mon[24786]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:28:21.747 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:21 smithi008 ceph-mon[24786]: Saving service prometheus spec with placement count:1 2024-04-14T13:28:21.747 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:21 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:21.747 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:21 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:22.187 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-04-14T13:28:22.720 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying node-exporter service with default placement... 2024-04-14T13:28:23.274 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:23 smithi008 ceph-mon[24786]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:28:23.275 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:23 smithi008 ceph-mon[24786]: Saving service grafana spec with placement count:1 2024-04-14T13:28:23.275 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:23 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:23.323 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-04-14T13:28:24.029 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying alertmanager service with default placement... 2024-04-14T13:28:24.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:24 smithi008 ceph-mon[24786]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:28:24.413 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:24 smithi008 ceph-mon[24786]: Saving service node-exporter spec with placement * 2024-04-14T13:28:24.413 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:24 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:24.644 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-04-14T13:28:25.755 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:25 smithi008 ceph-mon[24786]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:28:25.755 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:25 smithi008 ceph-mon[24786]: Saving service alertmanager spec with placement count:1 2024-04-14T13:28:25.755 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:25 smithi008 ceph-mon[24786]: from='mgr.14120 172.21.15.8:0/3510634276' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:27.322 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:27 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2056954115' entity='client.admin' 2024-04-14T13:28:27.322 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:27 smithi008 ceph-mon[24786]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:28:28.026 INFO:teuthology.orchestra.run.smithi008.stderr:Enabling the dashboard module... 2024-04-14T13:28:28.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:28 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/418882678' entity='client.admin' 2024-04-14T13:28:29.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:29 smithi008 ceph-mon[24786]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:28:29.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:29 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/53933304' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-14T13:28:30.537 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:30 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/53933304' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-14T13:28:30.538 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:30 smithi008 ceph-mon[24786]: mgrmap e9: smithi008.yemlhr(active, since 23s) 2024-04-14T13:28:30.617 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-04-14T13:28:30.618 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-14T13:28:30.618 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "available": true, 2024-04-14T13:28:30.618 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "active_name": "smithi008.yemlhr", 2024-04-14T13:28:30.618 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-14T13:28:30.618 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-04-14T13:28:31.240 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for the mgr to restart... 2024-04-14T13:28:31.240 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for mgr epoch 9... 2024-04-14T13:28:31.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:31 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2726603084' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-14T13:28:36.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:36 smithi008 ceph-mon[24786]: Active manager daemon smithi008.yemlhr restarted 2024-04-14T13:28:36.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:36 smithi008 ceph-mon[24786]: Activating manager daemon smithi008.yemlhr 2024-04-14T13:28:36.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:36 smithi008 ceph-mon[24786]: osdmap e3: 0 total, 0 up, 0 in 2024-04-14T13:28:36.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:36 smithi008 ceph-mon[24786]: mgrmap e10: smithi008.yemlhr(active, starting, since 0.0529071s) 2024-04-14T13:28:36.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:36 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:28:36.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:36 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr metadata", "who": "smithi008.yemlhr", "id": "smithi008.yemlhr"}]: dispatch 2024-04-14T13:28:36.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:36 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T13:28:36.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:36 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T13:28:36.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:36 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T13:28:36.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:36 smithi008 ceph-mon[24786]: Manager daemon smithi008.yemlhr is now available 2024-04-14T13:28:36.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:36 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:28:36.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:36 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:28:36.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:36 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:28:36.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:36 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:28:36.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:36 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:36.369 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-04-14T13:28:36.369 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-14T13:28:36.369 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "initialized": true 2024-04-14T13:28:36.369 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-04-14T13:28:37.049 INFO:teuthology.orchestra.run.smithi008.stderr:mgr epoch 9 is available 2024-04-14T13:28:37.050 INFO:teuthology.orchestra.run.smithi008.stderr:Generating a dashboard self-signed certificate... 2024-04-14T13:28:37.557 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:37 smithi008 ceph-mon[24786]: [14/Apr/2024:13:28:35] ENGINE Bus STARTING 2024-04-14T13:28:37.557 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:37 smithi008 ceph-mon[24786]: [14/Apr/2024:13:28:35] ENGINE Serving on https://172.21.15.8:7150 2024-04-14T13:28:37.558 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:37 smithi008 ceph-mon[24786]: [14/Apr/2024:13:28:35] ENGINE Bus STARTED 2024-04-14T13:28:37.558 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:37 smithi008 ceph-mon[24786]: mgrmap e11: smithi008.yemlhr(active, since 1.05557s) 2024-04-14T13:28:37.827 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-14T13:28:38.429 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:38 smithi008 ceph-mon[24786]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-14T13:28:38.429 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:38 smithi008 ceph-mon[24786]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-14T13:28:38.429 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:38 smithi008 ceph-mon[24786]: mgrmap e12: smithi008.yemlhr(active, since 2s) 2024-04-14T13:28:38.429 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:38 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:38.429 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:38 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:38.429 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:38 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:38.431 INFO:teuthology.orchestra.run.smithi008.stderr:Creating initial admin user... 2024-04-14T13:28:39.385 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$i3kOEnmEawuzoLS0nrabKua5G54UkxBoRu1hzQTezQBVVnAswi3jG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713101319, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-14T13:28:39.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:39 smithi008 ceph-mon[24786]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:28:39.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:39 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:40.734 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:40 smithi008 ceph-mon[24786]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:28:40.735 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:40 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:40.736 INFO:teuthology.orchestra.run.smithi008.stderr:Fetching dashboard port number... 2024-04-14T13:28:41.389 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 8443 2024-04-14T13:28:41.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:41 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/1002430172' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-14T13:28:42.003 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: disabled 2024-04-14T13:28:42.011 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: inactive 2024-04-14T13:28:42.011 INFO:teuthology.orchestra.run.smithi008.stderr:firewalld.service is not enabled 2024-04-14T13:28:42.011 INFO:teuthology.orchestra.run.smithi008.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-14T13:28:42.013 INFO:teuthology.orchestra.run.smithi008.stderr:Ceph Dashboard is now available at: 2024-04-14T13:28:42.014 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-14T13:28:42.014 INFO:teuthology.orchestra.run.smithi008.stderr: URL: https://smithi008.front.sepia.ceph.com:8443/ 2024-04-14T13:28:42.014 INFO:teuthology.orchestra.run.smithi008.stderr: User: admin 2024-04-14T13:28:42.014 INFO:teuthology.orchestra.run.smithi008.stderr: Password: pa3kp6x6k7 2024-04-14T13:28:42.014 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-14T13:28:42.014 INFO:teuthology.orchestra.run.smithi008.stderr:Enabling autotune for osd_memory_target 2024-04-14T13:28:42.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:42 smithi008 ceph-mon[24786]: mgrmap e13: smithi008.yemlhr(active, since 6s) 2024-04-14T13:28:44.077 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-14T13:28:44.675 INFO:teuthology.orchestra.run.smithi008.stderr:You can access the Ceph CLI with: 2024-04-14T13:28:44.675 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-14T13:28:44.675 INFO:teuthology.orchestra.run.smithi008.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-14T13:28:44.675 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-14T13:28:44.675 INFO:teuthology.orchestra.run.smithi008.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-14T13:28:44.675 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-14T13:28:44.676 INFO:teuthology.orchestra.run.smithi008.stderr: ceph telemetry on 2024-04-14T13:28:44.676 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-14T13:28:44.676 INFO:teuthology.orchestra.run.smithi008.stderr:For more information see: 2024-04-14T13:28:44.676 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-14T13:28:44.676 INFO:teuthology.orchestra.run.smithi008.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-14T13:28:44.676 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-14T13:28:44.676 INFO:teuthology.orchestra.run.smithi008.stderr:Bootstrap complete. 2024-04-14T13:28:44.722 INFO:tasks.cephadm:Fetching config... 2024-04-14T13:28:44.722 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-14T13:28:44.722 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-14T13:28:44.738 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-14T13:28:44.738 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-14T13:28:44.738 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-14T13:28:44.794 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-14T13:28:44.794 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-14T13:28:44.794 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/keyring of=/dev/stdout 2024-04-14T13:28:44.862 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-14T13:28:44.863 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-14T13:28:44.863 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-14T13:28:44.918 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-14T13:28:44.918 DEBUG:teuthology.orchestra.run.smithi008:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/pU9P0fE4tzIbdwggbfTUC2texBubxDHJELjKn1Th2+S/W17ZqXwNMpSj0AvFkmXN/EN6ouVcbPGCZ6wvxTGiV9qHA8dPCqlY5ebJhJmijHUsGyLX3lOXZqwTrA/8MFg2jA9clYgzzCHYe1Vqh1wbts6/BT/KDZ+eDk0r3fB9EWyLwZL9dRN1V4TkI8a+v2dUISZniiJK0g6FUODFsCclowARL75HcaZ29MRl4dkXni8/tRv8dZzuZ3x9OZjwb4pJ8tg8z4I0q4uZGIGw1IF9c8zTyQPVkYzn26DJ5GYAwL6Z+BpRA476EAarVLah8IamWoZdHmNN3/tWLrnzw/9b0Wy5X/dlMCtB/DuZdCSeD3DzaYE06Fdis5NvACpCSS+T1FRrn63fwi8angHzvBLeWzAratVJdn+ISySkDgW3PH0dXJ/9KMwJjIY27s3w2V0QnoFvJd03fiJa0c97/hENn0cg5+CYa7QIawZPKHT4tXM7F+m624bEp+Su3P7/SFU= ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-14T13:28:45.009 INFO:teuthology.orchestra.run.smithi008.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/pU9P0fE4tzIbdwggbfTUC2texBubxDHJELjKn1Th2+S/W17ZqXwNMpSj0AvFkmXN/EN6ouVcbPGCZ6wvxTGiV9qHA8dPCqlY5ebJhJmijHUsGyLX3lOXZqwTrA/8MFg2jA9clYgzzCHYe1Vqh1wbts6/BT/KDZ+eDk0r3fB9EWyLwZL9dRN1V4TkI8a+v2dUISZniiJK0g6FUODFsCclowARL75HcaZ29MRl4dkXni8/tRv8dZzuZ3x9OZjwb4pJ8tg8z4I0q4uZGIGw1IF9c8zTyQPVkYzn26DJ5GYAwL6Z+BpRA476EAarVLah8IamWoZdHmNN3/tWLrnzw/9b0Wy5X/dlMCtB/DuZdCSeD3DzaYE06Fdis5NvACpCSS+T1FRrn63fwi8angHzvBLeWzAratVJdn+ISySkDgW3PH0dXJ/9KMwJjIY27s3w2V0QnoFvJd03fiJa0c97/hENn0cg5+CYa7QIawZPKHT4tXM7F+m624bEp+Su3P7/SFU= ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:28:45.025 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/pU9P0fE4tzIbdwggbfTUC2texBubxDHJELjKn1Th2+S/W17ZqXwNMpSj0AvFkmXN/EN6ouVcbPGCZ6wvxTGiV9qHA8dPCqlY5ebJhJmijHUsGyLX3lOXZqwTrA/8MFg2jA9clYgzzCHYe1Vqh1wbts6/BT/KDZ+eDk0r3fB9EWyLwZL9dRN1V4TkI8a+v2dUISZniiJK0g6FUODFsCclowARL75HcaZ29MRl4dkXni8/tRv8dZzuZ3x9OZjwb4pJ8tg8z4I0q4uZGIGw1IF9c8zTyQPVkYzn26DJ5GYAwL6Z+BpRA476EAarVLah8IamWoZdHmNN3/tWLrnzw/9b0Wy5X/dlMCtB/DuZdCSeD3DzaYE06Fdis5NvACpCSS+T1FRrn63fwi8angHzvBLeWzAratVJdn+ISySkDgW3PH0dXJ/9KMwJjIY27s3w2V0QnoFvJd03fiJa0c97/hENn0cg5+CYa7QIawZPKHT4tXM7F+m624bEp+Su3P7/SFU= ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-14T13:28:45.072 INFO:teuthology.orchestra.run.smithi012.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/pU9P0fE4tzIbdwggbfTUC2texBubxDHJELjKn1Th2+S/W17ZqXwNMpSj0AvFkmXN/EN6ouVcbPGCZ6wvxTGiV9qHA8dPCqlY5ebJhJmijHUsGyLX3lOXZqwTrA/8MFg2jA9clYgzzCHYe1Vqh1wbts6/BT/KDZ+eDk0r3fB9EWyLwZL9dRN1V4TkI8a+v2dUISZniiJK0g6FUODFsCclowARL75HcaZ29MRl4dkXni8/tRv8dZzuZ3x9OZjwb4pJ8tg8z4I0q4uZGIGw1IF9c8zTyQPVkYzn26DJ5GYAwL6Z+BpRA476EAarVLah8IamWoZdHmNN3/tWLrnzw/9b0Wy5X/dlMCtB/DuZdCSeD3DzaYE06Fdis5NvACpCSS+T1FRrn63fwi8angHzvBLeWzAratVJdn+ISySkDgW3PH0dXJ/9KMwJjIY27s3w2V0QnoFvJd03fiJa0c97/hENn0cg5+CYa7QIawZPKHT4tXM7F+m624bEp+Su3P7/SFU= ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:28:45.087 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-14T13:28:45.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:45 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/121693392' entity='client.admin' 2024-04-14T13:28:49.382 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:49 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:49.383 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:49 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi008", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:28:49.383 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:49 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:49.383 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:49 smithi008 ceph-mon[24786]: Deploying daemon alertmanager.smithi008 on smithi008 2024-04-14T13:28:49.383 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:49 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/1476333586' entity='client.admin' 2024-04-14T13:28:49.406 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-14T13:28:49.406 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-14T13:28:53.528 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:53 smithi008 ceph-mon[24786]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:28:53.528 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:53 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:53.553 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi012 2024-04-14T13:28:53.553 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-14T13:28:53.553 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/etc/ceph/ceph.conf 2024-04-14T13:28:53.571 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-14T13:28:53.571 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:28:53.627 INFO:tasks.cephadm:Adding host smithi012 to orchestrator... 2024-04-14T13:28:53.628 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph orch host add smithi012 2024-04-14T13:28:55.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:55 smithi008 ceph-mon[24786]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:28:57.585 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:57 smithi008 ceph-mon[24786]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:28:58.371 INFO:teuthology.orchestra.run.smithi008.stdout:Added host 'smithi012' with addr '172.21.15.12' 2024-04-14T13:28:59.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:58 smithi008 ceph-mon[24786]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi012", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:28:59.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:58 smithi008 ceph-mon[24786]: Deploying cephadm binary to smithi012 2024-04-14T13:28:59.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:58 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:59.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:58 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-14T13:28:59.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:58 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-14T13:28:59.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:58 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:28:59.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:58 smithi008 ceph-mon[24786]: Deploying daemon crash.smithi008 on smithi008 2024-04-14T13:28:59.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:58 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:28:59.516 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph orch host ls --format=json 2024-04-14T13:29:00.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:59 smithi008 ceph-mon[24786]: Added host smithi012 2024-04-14T13:29:00.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:28:59 smithi008 ceph-mon[24786]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:29:01.759 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:29:01.759 INFO:teuthology.orchestra.run.smithi008.stdout:[{"addr": "172.21.15.8", "hostname": "smithi008", "labels": [], "status": ""}, {"addr": "172.21.15.12", "hostname": "smithi012", "labels": [], "status": ""}] 2024-04-14T13:29:01.759 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:01 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:01.759 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:01 smithi008 ceph-mon[24786]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:29:02.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:02 smithi008 ceph-mon[24786]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:29:02.686 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-14T13:29:02.686 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph osd crush tunables default 2024-04-14T13:29:04.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:03 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:04.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:03 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:04.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:03 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:04.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:03 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T13:29:04.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:03 smithi008 ceph-mon[24786]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T13:29:04.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:03 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:04.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:03 smithi008 ceph-mon[24786]: Deploying daemon grafana.smithi008 on smithi008 2024-04-14T13:29:04.073 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:03 smithi008 ceph-mon[24786]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:29:05.030 INFO:teuthology.orchestra.run.smithi008.stderr:adjusted tunables profile to default 2024-04-14T13:29:05.329 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:05 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2512904348' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-14T13:29:05.666 INFO:tasks.cephadm:Adding mon.smithi008 on smithi008 2024-04-14T13:29:05.666 INFO:tasks.cephadm:Adding mon.smithi012 on smithi012 2024-04-14T13:29:05.666 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph orch apply mon '2;smithi008:172.21.15.8=smithi008;smithi012:172.21.15.12=smithi012' 2024-04-14T13:29:06.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:06 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2512904348' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-14T13:29:06.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:06 smithi008 ceph-mon[24786]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T13:29:06.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:06 smithi008 ceph-mon[24786]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:29:06.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:06 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:07.445 INFO:teuthology.orchestra.run.smithi012.stdout:Scheduled mon update... 2024-04-14T13:29:07.707 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:07 smithi008 ceph-mon[24786]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:29:08.016 DEBUG:teuthology.orchestra.run.smithi012:mon.smithi012> sudo journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@mon.smithi012.service 2024-04-14T13:29:08.019 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-14T13:29:08.019 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T13:29:08.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:08 smithi008 ceph-mon[24786]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi008:172.21.15.8=smithi008;smithi012:172.21.15.12=smithi012", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:29:08.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:08 smithi008 ceph-mon[24786]: Saving service mon spec with placement smithi008:172.21.15.8=smithi008;smithi012:172.21.15.12=smithi012;count:2 2024-04-14T13:29:08.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:08 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:09.752 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:09 smithi008 ceph-mon[24786]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:29:09.910 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:29:09.910 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":1,"fsid":"a8b03f6a-fa62-11ee-bc8e-c7b262605968","modified":"2024-04-14T13:27:27.599436Z","created":"2024-04-14T13:27:27.599436Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-14T13:29:09.912 INFO:teuthology.orchestra.run.smithi012.stderr:dumped monmap epoch 1 2024-04-14T13:29:10.767 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:10 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/1842021616' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:29:11.459 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-14T13:29:11.459 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T13:29:11.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:11 smithi008 ceph-mon[24786]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:29:13.318 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:29:13.318 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":1,"fsid":"a8b03f6a-fa62-11ee-bc8e-c7b262605968","modified":"2024-04-14T13:27:27.599436Z","created":"2024-04-14T13:27:27.599436Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-14T13:29:13.319 INFO:teuthology.orchestra.run.smithi012.stderr:dumped monmap epoch 1 2024-04-14T13:29:13.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:13 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/2066427982' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:29:14.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:14 smithi008 ceph-mon[24786]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:29:15.015 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-14T13:29:15.015 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T13:29:16.825 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:29:16.825 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":1,"fsid":"a8b03f6a-fa62-11ee-bc8e-c7b262605968","modified":"2024-04-14T13:27:27.599436Z","created":"2024-04-14T13:27:27.599436Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-14T13:29:16.826 INFO:teuthology.orchestra.run.smithi012.stderr:dumped monmap epoch 1 2024-04-14T13:29:16.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:16 smithi008 ceph-mon[24786]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:29:17.777 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:17 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/3879849102' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:29:18.433 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-14T13:29:18.434 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T13:29:18.742 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:18 smithi008 ceph-mon[24786]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:29:19.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:19 smithi008 ceph-mon[24786]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:29:20.335 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:29:20.335 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":1,"fsid":"a8b03f6a-fa62-11ee-bc8e-c7b262605968","modified":"2024-04-14T13:27:27.599436Z","created":"2024-04-14T13:27:27.599436Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-14T13:29:20.337 INFO:teuthology.orchestra.run.smithi012.stderr:dumped monmap epoch 1 2024-04-14T13:29:20.469 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:20 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/2582431854' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:29:21.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:21 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:21.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:21 smithi008 ceph-mon[24786]: Deploying daemon node-exporter.smithi008 on smithi008 2024-04-14T13:29:21.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:21 smithi008 ceph-mon[24786]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:29:21.892 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-14T13:29:21.892 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T13:29:23.756 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:29:23.757 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":1,"fsid":"a8b03f6a-fa62-11ee-bc8e-c7b262605968","modified":"2024-04-14T13:27:27.599436Z","created":"2024-04-14T13:27:27.599436Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-14T13:29:23.758 INFO:teuthology.orchestra.run.smithi012.stderr:dumped monmap epoch 1 2024-04-14T13:29:23.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:23 smithi008 ceph-mon[24786]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:29:24.676 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:24 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/883874354' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:29:25.269 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-14T13:29:25.270 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T13:29:26.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:25 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:26.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:25 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-14T13:29:26.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:25 smithi008 ceph-mon[24786]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:29:26.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:25 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:27.054 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:26 smithi008 ceph-mon[24786]: from='mgr.14162 172.21.15.8:0/1546927539' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-14T13:29:27.054 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:26 smithi008 ceph-mon[24786]: mgrmap e14: smithi008.yemlhr(active, since 50s) 2024-04-14T13:29:27.084 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:29:27.084 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":1,"fsid":"a8b03f6a-fa62-11ee-bc8e-c7b262605968","modified":"2024-04-14T13:27:27.599436Z","created":"2024-04-14T13:27:27.599436Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-14T13:29:27.087 INFO:teuthology.orchestra.run.smithi012.stderr:dumped monmap epoch 1 2024-04-14T13:29:28.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:27 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/2477794868' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:29:28.678 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-14T13:29:28.679 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T13:29:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:29:30.442 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":1,"fsid":"a8b03f6a-fa62-11ee-bc8e-c7b262605968","modified":"2024-04-14T13:27:27.599436Z","created":"2024-04-14T13:27:27.599436Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-14T13:29:30.444 INFO:teuthology.orchestra.run.smithi012.stderr:dumped monmap epoch 1 2024-04-14T13:29:30.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:30 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/1714578811' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:29:32.020 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-14T13:29:32.020 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T13:29:33.024 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:32 smithi008 ceph-mon[24786]: Active manager daemon smithi008.yemlhr restarted 2024-04-14T13:29:33.024 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:32 smithi008 ceph-mon[24786]: Activating manager daemon smithi008.yemlhr 2024-04-14T13:29:33.024 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:32 smithi008 ceph-mon[24786]: osdmap e5: 0 total, 0 up, 0 in 2024-04-14T13:29:33.024 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:32 smithi008 ceph-mon[24786]: mgrmap e15: smithi008.yemlhr(active, starting, since 0.0524233s) 2024-04-14T13:29:33.024 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:29:33.024 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr metadata", "who": "smithi008.yemlhr", "id": "smithi008.yemlhr"}]: dispatch 2024-04-14T13:29:33.024 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T13:29:33.025 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T13:29:33.025 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T13:29:33.025 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:32 smithi008 ceph-mon[24786]: Manager daemon smithi008.yemlhr is now available 2024-04-14T13:29:33.025 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:29:33.025 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:29:33.025 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:29:33.025 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:29:33.025 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:33.025 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:29:33.025 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:29:33.810 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:29:33.810 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":1,"fsid":"a8b03f6a-fa62-11ee-bc8e-c7b262605968","modified":"2024-04-14T13:27:27.599436Z","created":"2024-04-14T13:27:27.599436Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-14T13:29:33.811 INFO:teuthology.orchestra.run.smithi012.stderr:dumped monmap epoch 1 2024-04-14T13:29:33.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:33 smithi008 ceph-mon[24786]: [14/Apr/2024:13:29:32] ENGINE Bus STARTING 2024-04-14T13:29:33.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:33 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:33.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:33 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:33.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:33 smithi008 ceph-mon[24786]: mgrmap e16: smithi008.yemlhr(active, since 1.05496s) 2024-04-14T13:29:33.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:33 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:35.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:34 smithi008 ceph-mon[24786]: [14/Apr/2024:13:29:32] ENGINE Serving on https://172.21.15.8:7150 2024-04-14T13:29:35.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:34 smithi008 ceph-mon[24786]: [14/Apr/2024:13:29:32] ENGINE Bus STARTED 2024-04-14T13:29:35.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:34 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/2191385715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:29:35.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:34 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:35.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:34 smithi008 ceph-mon[24786]: mgrmap e17: smithi008.yemlhr(active, since 2s) 2024-04-14T13:29:35.321 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-14T13:29:35.321 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T13:29:37.806 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:29:37.807 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":1,"fsid":"a8b03f6a-fa62-11ee-bc8e-c7b262605968","modified":"2024-04-14T13:27:27.599436Z","created":"2024-04-14T13:27:27.599436Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-14T13:29:37.808 INFO:teuthology.orchestra.run.smithi012.stderr:dumped monmap epoch 1 2024-04-14T13:29:38.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:37 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/2284442100' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:29:39.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:39.510 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:39.510 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi012", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:29:39.721 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-14T13:29:39.721 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T13:29:41.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:40 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:41.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:40 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:41.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:40 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi008", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:29:41.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:40 smithi008 ceph-mon[24786]: Updating smithi008:/etc/ceph/ceph.conf 2024-04-14T13:29:41.756 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:29:41.756 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":1,"fsid":"a8b03f6a-fa62-11ee-bc8e-c7b262605968","modified":"2024-04-14T13:27:27.599436Z","created":"2024-04-14T13:27:27.599436Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-14T13:29:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:dumped monmap epoch 1 2024-04-14T13:29:42.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:42 smithi008 ceph-mon[24786]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:29:42.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:42 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:42.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:42 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:42.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:42 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi012", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-14T13:29:42.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:42 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi012", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-14T13:29:42.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:42 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:29:42.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:42 smithi008 ceph-mon[24786]: Deploying daemon crash.smithi012 on smithi012 2024-04-14T13:29:42.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:42 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/339656864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:29:44.413 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-14T13:29:44.414 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T13:29:45.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:45 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:45.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:45 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi012.ksvnxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:29:45.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:45 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi012.ksvnxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-14T13:29:45.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:45 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:29:45.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:45 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:29:45.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:45 smithi008 ceph-mon[24786]: Deploying daemon mgr.smithi012.ksvnxm on smithi012 2024-04-14T13:29:47.305 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:29:47.305 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":1,"fsid":"a8b03f6a-fa62-11ee-bc8e-c7b262605968","modified":"2024-04-14T13:27:27.599436Z","created":"2024-04-14T13:27:27.599436Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-14T13:29:47.307 INFO:teuthology.orchestra.run.smithi012.stderr:dumped monmap epoch 1 2024-04-14T13:29:48.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:47 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:48.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:47 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/558577397' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:29:48.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:47 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:48.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:47 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:29:48.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:47 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:29:48.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:47 smithi008 ceph-mon[24786]: Deploying daemon mon.smithi012 on smithi012 2024-04-14T13:29:49.079 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-14T13:29:49.079 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T13:29:52.059 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:29:52.060 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":1,"fsid":"a8b03f6a-fa62-11ee-bc8e-c7b262605968","modified":"2024-04-14T13:27:27.599436Z","created":"2024-04-14T13:27:27.599436Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-14T13:29:52.061 INFO:teuthology.orchestra.run.smithi012.stderr:dumped monmap epoch 1 2024-04-14T13:29:53.991 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-14T13:29:53.991 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T13:29:57.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:29:57.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:29:57.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:29:57.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: mon.smithi008 calling monitor election 2024-04-14T13:29:57.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:29:57.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:29:57.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.12:0/2805406185' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi012.ksvnxm/crt"}]: dispatch 2024-04-14T13:29:57.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:29:57.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:29:57.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: mon.smithi012 calling monitor election 2024-04-14T13:29:57.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:29:57.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:29:57.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:29:57.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:29:57.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: mon.smithi008 is new leader, mons smithi008,smithi012 in quorum (ranks 0,1) 2024-04-14T13:29:57.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: monmap e2: 2 mons at {smithi008=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0],smithi012=[v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0]} 2024-04-14T13:29:57.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: fsmap 2024-04-14T13:29:57.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: osdmap e5: 0 total, 0 up, 0 in 2024-04-14T13:29:57.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: mgrmap e17: smithi008.yemlhr(active, since 25s) 2024-04-14T13:29:57.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: Standby manager daemon smithi012.ksvnxm started 2024-04-14T13:29:57.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.12:0/2805406185' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T13:29:57.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: overall HEALTH_OK 2024-04-14T13:29:57.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.12:0/2805406185' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi012.ksvnxm/key"}]: dispatch 2024-04-14T13:29:57.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:29:57.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:57 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.12:0/2805406185' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T13:29:58.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:58 smithi008 ceph-mon[24786]: Deploying daemon node-exporter.smithi012 on smithi012 2024-04-14T13:29:58.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:58 smithi008 ceph-mon[24786]: mgrmap e18: smithi008.yemlhr(active, since 25s), standbys: smithi012.ksvnxm 2024-04-14T13:29:58.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:58 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr metadata", "who": "smithi012.ksvnxm", "id": "smithi012.ksvnxm"}]: dispatch 2024-04-14T13:29:58.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:58 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:29:58.632 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:29:58 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:29:58.964 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-14T13:29:58.964 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":2,"fsid":"a8b03f6a-fa62-11ee-bc8e-c7b262605968","modified":"2024-04-14T13:29:52.242401Z","created":"2024-04-14T13:27:27.599436Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi012","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-14T13:29:58.965 INFO:teuthology.orchestra.run.smithi012.stderr:dumped monmap epoch 2 2024-04-14T13:29:59.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:59 smithi008 ceph-mon[24786]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:29:59.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:29:59 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/2367952707' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:29:59.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:29:59 smithi012 ceph-mon[27236]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:29:59.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:29:59 smithi012 ceph-mon[27236]: from='client.? 172.21.15.12:0/2367952707' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:29:59.917 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-14T13:29:59.917 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph config generate-minimal-conf 2024-04-14T13:30:00.516 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:00 smithi008 ceph-mon[24786]: overall HEALTH_OK 2024-04-14T13:30:00.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:00 smithi012 ceph-mon[27236]: overall HEALTH_OK 2024-04-14T13:30:01.788 INFO:teuthology.orchestra.run.smithi008.stdout:# minimal ceph.conf for a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:30:01.788 INFO:teuthology.orchestra.run.smithi008.stdout:[global] 2024-04-14T13:30:01.788 INFO:teuthology.orchestra.run.smithi008.stdout: fsid = a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:30:01.788 INFO:teuthology.orchestra.run.smithi008.stdout: mon_host = [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-04-14T13:30:01.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:01 smithi008 ceph-mon[24786]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:01.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:01 smithi012 ceph-mon[27236]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:02.376 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-14T13:30:02.377 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-14T13:30:02.377 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/ceph/ceph.conf 2024-04-14T13:30:02.408 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-14T13:30:02.408 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:30:02.476 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-14T13:30:02.476 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.conf 2024-04-14T13:30:02.510 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-14T13:30:02.510 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:30:02.579 INFO:tasks.cephadm:Deploying OSDs... 2024-04-14T13:30:02.580 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-14T13:30:02.580 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/scratch_devs of=/dev/stdout 2024-04-14T13:30:02.595 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-14T13:30:02.595 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_1 2024-04-14T13:30:02.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:02 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2954616665' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:02.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:02 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:02.615 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-14T13:30:02.615 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T13:30:02.615 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-14T13:30:02.615 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T13:30:02.615 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T13:30:02.615 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-04-14 13:29:21.184899411 +0000 2024-04-14T13:30:02.615 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-04-14 13:26:29.803226430 +0000 2024-04-14T13:30:02.615 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-04-14 13:26:29.803226430 +0000 2024-04-14T13:30:02.615 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-04-14 13:26:29.803226430 +0000 2024-04-14T13:30:02.616 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-14T13:30:02.680 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-04-14T13:30:02.680 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-04-14T13:30:02.681 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 0.000117451 s, 4.4 MB/s 2024-04-14T13:30:02.682 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-14T13:30:02.738 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_2 2024-04-14T13:30:02.792 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-14T13:30:02.792 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T13:30:02.792 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-04-14T13:30:02.793 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T13:30:02.793 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T13:30:02.793 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-04-14 13:29:21.185899401 +0000 2024-04-14T13:30:02.793 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-04-14 13:26:30.145222959 +0000 2024-04-14T13:30:02.793 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-04-14 13:26:30.145222959 +0000 2024-04-14T13:30:02.793 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-04-14 13:26:30.145222959 +0000 2024-04-14T13:30:02.793 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-14T13:30:02.798 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:02 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/2954616665' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:02.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:02 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:02.858 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-04-14T13:30:02.859 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-04-14T13:30:02.859 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 0.000102864 s, 5.0 MB/s 2024-04-14T13:30:02.860 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-14T13:30:02.915 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_3 2024-04-14T13:30:02.969 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-14T13:30:02.969 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T13:30:02.969 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-14T13:30:02.970 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T13:30:02.970 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T13:30:02.970 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-04-14 13:29:21.185899401 +0000 2024-04-14T13:30:02.970 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-04-14 13:26:30.487219487 +0000 2024-04-14T13:30:02.970 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-04-14 13:26:30.487219487 +0000 2024-04-14T13:30:02.970 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-04-14 13:26:30.487219487 +0000 2024-04-14T13:30:02.970 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-14T13:30:03.035 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-04-14T13:30:03.035 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-04-14T13:30:03.035 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 9.9479e-05 s, 5.1 MB/s 2024-04-14T13:30:03.037 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-14T13:30:03.092 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_4 2024-04-14T13:30:03.146 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-14T13:30:03.146 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T13:30:03.146 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-14T13:30:03.146 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T13:30:03.146 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T13:30:03.146 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-04-14 13:29:21.186899391 +0000 2024-04-14T13:30:03.146 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-04-14 13:26:30.818216127 +0000 2024-04-14T13:30:03.146 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-04-14 13:26:30.818216127 +0000 2024-04-14T13:30:03.146 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-04-14 13:26:30.818216127 +0000 2024-04-14T13:30:03.147 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-14T13:30:03.210 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-04-14T13:30:03.210 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-04-14T13:30:03.210 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 0.000104811 s, 4.9 MB/s 2024-04-14T13:30:03.214 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-14T13:30:03.269 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-14T13:30:03.270 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/scratch_devs of=/dev/stdout 2024-04-14T13:30:03.285 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-14T13:30:03.285 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_1 2024-04-14T13:30:03.341 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-14T13:30:03.341 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T13:30:03.341 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-04-14T13:30:03.341 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T13:30:03.342 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T13:30:03.342 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-04-14 13:29:57.862976554 +0000 2024-04-14T13:30:03.342 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-04-14 13:26:22.615502070 +0000 2024-04-14T13:30:03.342 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-04-14 13:26:22.615502070 +0000 2024-04-14T13:30:03.342 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-04-14 13:26:22.615502070 +0000 2024-04-14T13:30:03.342 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-14T13:30:03.407 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-04-14T13:30:03.407 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-04-14T13:30:03.407 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000210839 s, 2.4 MB/s 2024-04-14T13:30:03.408 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-14T13:30:03.465 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_2 2024-04-14T13:30:03.520 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-14T13:30:03.520 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T13:30:03.520 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-14T13:30:03.520 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T13:30:03.520 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T13:30:03.520 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-04-14 13:29:57.862976554 +0000 2024-04-14T13:30:03.520 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-04-14 13:26:22.983498758 +0000 2024-04-14T13:30:03.521 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-04-14 13:26:22.983498758 +0000 2024-04-14T13:30:03.521 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-04-14 13:26:22.983498758 +0000 2024-04-14T13:30:03.521 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-14T13:30:03.549 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:03 smithi012 ceph-mon[27236]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:03.573 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-04-14T13:30:03.573 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-04-14T13:30:03.574 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000191349 s, 2.7 MB/s 2024-04-14T13:30:03.575 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-14T13:30:03.630 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_3 2024-04-14T13:30:03.684 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-14T13:30:03.684 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T13:30:03.684 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-14T13:30:03.684 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T13:30:03.685 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T13:30:03.685 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-04-14 13:29:57.862976554 +0000 2024-04-14T13:30:03.685 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-04-14 13:26:23.317495752 +0000 2024-04-14T13:30:03.685 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-04-14 13:26:23.317495752 +0000 2024-04-14T13:30:03.685 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-04-14 13:26:23.317495752 +0000 2024-04-14T13:30:03.685 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-14T13:30:03.749 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-04-14T13:30:03.749 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-04-14T13:30:03.749 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000199401 s, 2.6 MB/s 2024-04-14T13:30:03.751 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-14T13:30:03.806 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_4 2024-04-14T13:30:03.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:03 smithi008 ceph-mon[24786]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:03.861 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-14T13:30:03.862 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T13:30:03.862 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-14T13:30:03.862 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T13:30:03.862 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T13:30:03.862 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-04-14 13:29:57.863976544 +0000 2024-04-14T13:30:03.862 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-04-14 13:26:23.668492592 +0000 2024-04-14T13:30:03.862 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-04-14 13:26:23.668492592 +0000 2024-04-14T13:30:03.862 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-04-14 13:26:23.668492592 +0000 2024-04-14T13:30:03.862 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-14T13:30:03.927 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-04-14T13:30:03.928 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-04-14T13:30:03.928 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000199056 s, 2.6 MB/s 2024-04-14T13:30:03.929 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-14T13:30:03.985 INFO:tasks.cephadm:Deploying osd.0 on smithi008 with /dev/vg_nvme/lv_4... 2024-04-14T13:30:03.985 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-14T13:30:05.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:05 smithi008 ceph-mon[24786]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:05.243 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:05 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:05.243 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:05 smithi008 ceph-mon[24786]: Deploying daemon prometheus.smithi008 on smithi008 2024-04-14T13:30:05.549 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:05 smithi012 ceph-mon[27236]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:05.549 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:05 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:05.549 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:05 smithi012 ceph-mon[27236]: Deploying daemon prometheus.smithi008 on smithi008 2024-04-14T13:30:05.973 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-14T13:30:05.974 INFO:teuthology.orchestra.run.smithi008.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-14T13:30:05.974 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10+0 records in 2024-04-14T13:30:05.974 INFO:teuthology.orchestra.run.smithi008.stderr:10+0 records out 2024-04-14T13:30:05.974 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00855509 s, 1.2 GB/s 2024-04-14T13:30:05.974 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping successful for: 2024-04-14T13:30:06.513 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi008:vg_nvme/lv_4 2024-04-14T13:30:07.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:07 smithi012 ceph-mon[27236]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:07.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:07 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:07.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:07 smithi008 ceph-mon[24786]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:07.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:07 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:09.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:09 smithi012 ceph-mon[27236]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:09.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:09 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:30:09.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:09 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:30:09.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:09 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:09.803 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:09 smithi008 ceph-mon[24786]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:09.804 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:09 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:30:09.804 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:09 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:30:09.804 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:09 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:10.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:10 smithi012 ceph-mon[27236]: from='client.14240 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:30:10.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:10 smithi008 ceph-mon[24786]: from='client.14240 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:30:11.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:11 smithi012 ceph-mon[27236]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:11.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:11 smithi008 ceph-mon[24786]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:13.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:13 smithi012 ceph-mon[27236]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:13.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:13 smithi008 ceph-mon[24786]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:15.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:15 smithi012 ceph-mon[27236]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:15.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:15 smithi008 ceph-mon[24786]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:17.770 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:17 smithi008 ceph-mon[24786]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:17.781 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:17 smithi012 ceph-mon[27236]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:19.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:19 smithi008 ceph-mon[24786]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:19.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:19 smithi012 ceph-mon[27236]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:21.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:21 smithi008 ceph-mon[24786]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:21.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:21 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:21.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:21 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:30:21.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:21 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:21.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:21 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:30:21.879 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:21 smithi012 ceph-mon[27236]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:21.879 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:21 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:21.879 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:21 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:30:21.879 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:21 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:21.879 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:21 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:30:22.836 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:22 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/3125875260' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4967d1e8-5e91-4e77-b6eb-28d0d9af8475"}]: dispatch 2024-04-14T13:30:22.836 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:22 smithi012 ceph-mon[27236]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4967d1e8-5e91-4e77-b6eb-28d0d9af8475"}]: dispatch 2024-04-14T13:30:22.836 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:22 smithi012 ceph-mon[27236]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4967d1e8-5e91-4e77-b6eb-28d0d9af8475"}]': finished 2024-04-14T13:30:22.836 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:22 smithi012 ceph-mon[27236]: osdmap e6: 1 total, 0 up, 1 in 2024-04-14T13:30:22.837 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:22 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:30:22.837 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:22 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:22.837 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:22 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/2160917332' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:30:22.837 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:22 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:22.996 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:22 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/3125875260' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4967d1e8-5e91-4e77-b6eb-28d0d9af8475"}]: dispatch 2024-04-14T13:30:22.996 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:22 smithi008 ceph-mon[24786]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4967d1e8-5e91-4e77-b6eb-28d0d9af8475"}]: dispatch 2024-04-14T13:30:22.996 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:22 smithi008 ceph-mon[24786]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4967d1e8-5e91-4e77-b6eb-28d0d9af8475"}]': finished 2024-04-14T13:30:22.996 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:22 smithi008 ceph-mon[24786]: osdmap e6: 1 total, 0 up, 1 in 2024-04-14T13:30:22.996 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:22 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:30:22.996 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:22 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:22.996 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:22 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2160917332' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:30:22.997 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:22 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:23.889 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:23 smithi008 ceph-mon[24786]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:23.889 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:23 smithi008 ceph-mon[24786]: Updating smithi012:/etc/ceph/ceph.conf 2024-04-14T13:30:23.889 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:23 smithi008 ceph-mon[24786]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:30:23.889 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:23 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:24.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:23 smithi012 ceph-mon[27236]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:24.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:23 smithi012 ceph-mon[27236]: Updating smithi012:/etc/ceph/ceph.conf 2024-04-14T13:30:24.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:23 smithi012 ceph-mon[27236]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:30:24.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:23 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:26.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:25 smithi012 ceph-mon[27236]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:26.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:25 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:26.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:25 smithi012 ceph-mon[27236]: Updating smithi008:/etc/ceph/ceph.conf 2024-04-14T13:30:26.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:25 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:26.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:25 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:26.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:25 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:30:26.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:25 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:30:26.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:25 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:26.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:25 smithi008 ceph-mon[24786]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:26.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:25 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:26.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:25 smithi008 ceph-mon[24786]: Updating smithi008:/etc/ceph/ceph.conf 2024-04-14T13:30:26.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:25 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:26.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:25 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:26.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:25 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:30:26.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:25 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:30:26.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:25 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:27.047 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:26 smithi008 ceph-mon[24786]: Reconfiguring mon.smithi008 (unknown last config time)... 2024-04-14T13:30:27.047 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:26 smithi008 ceph-mon[24786]: Reconfiguring daemon mon.smithi008 on smithi008 2024-04-14T13:30:27.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:26 smithi012 ceph-mon[27236]: Reconfiguring mon.smithi008 (unknown last config time)... 2024-04-14T13:30:27.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:26 smithi012 ceph-mon[27236]: Reconfiguring daemon mon.smithi008 on smithi008 2024-04-14T13:30:28.025 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:27 smithi008 ceph-mon[24786]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:28.026 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:27 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:28.026 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:27 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.yemlhr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:30:28.026 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:27 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:30:28.026 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:27 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:28.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:27 smithi012 ceph-mon[27236]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:28.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:27 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:28.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:27 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.yemlhr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:30:28.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:27 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:30:28.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:27 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:28.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:28 smithi008 ceph-mon[24786]: Reconfiguring mgr.smithi008.yemlhr (unknown last config time)... 2024-04-14T13:30:28.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:28 smithi008 ceph-mon[24786]: Reconfiguring daemon mgr.smithi008.yemlhr on smithi008 2024-04-14T13:30:28.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:28 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T13:30:28.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:28 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:29.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:28 smithi012 ceph-mon[27236]: Reconfiguring mgr.smithi008.yemlhr (unknown last config time)... 2024-04-14T13:30:29.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:28 smithi012 ceph-mon[27236]: Reconfiguring daemon mgr.smithi008.yemlhr on smithi008 2024-04-14T13:30:29.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:28 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T13:30:29.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:28 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:29.850 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:29 smithi008 ceph-mon[24786]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:29.850 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:29 smithi008 ceph-mon[24786]: Deploying daemon osd.0 on smithi008 2024-04-14T13:30:29.850 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:29 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:30.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:29 smithi012 ceph-mon[27236]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:30.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:29 smithi012 ceph-mon[27236]: Deploying daemon osd.0 on smithi008 2024-04-14T13:30:30.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:29 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:30.895 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:30 smithi008 ceph-mon[24786]: Reconfiguring alertmanager.smithi008 (dependencies changed)... 2024-04-14T13:30:30.895 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:30 smithi008 ceph-mon[24786]: Reconfiguring daemon alertmanager.smithi008 on smithi008 2024-04-14T13:30:31.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:30 smithi012 ceph-mon[27236]: Reconfiguring alertmanager.smithi008 (dependencies changed)... 2024-04-14T13:30:31.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:30 smithi012 ceph-mon[27236]: Reconfiguring daemon alertmanager.smithi008 on smithi008 2024-04-14T13:30:32.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:31 smithi008 ceph-mon[24786]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:32.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:31 smithi012 ceph-mon[27236]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:33.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:30:33.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:30:33.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:30:33.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:30:34.070 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:33 smithi008 ceph-mon[24786]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:34.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:33 smithi012 ceph-mon[27236]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:35.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:35 smithi008 ceph-mon[24786]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:35.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:35 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:36.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:35 smithi012 ceph-mon[27236]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:36.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:35 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:37.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:37 smithi008 ceph-mon[24786]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:38.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:37 smithi012 ceph-mon[27236]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:38.844 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:38 smithi008 ceph-mon[24786]: from='osd.0 [v2:172.21.15.8:6802/1782930059,v1:172.21.15.8:6803/1782930059]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T13:30:39.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:38 smithi012 ceph-mon[27236]: from='osd.0 [v2:172.21.15.8:6802/1782930059,v1:172.21.15.8:6803/1782930059]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T13:30:39.768 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:39 smithi008 ceph-mon[24786]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:39.768 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:39 smithi008 ceph-mon[24786]: from='osd.0 [v2:172.21.15.8:6802/1782930059,v1:172.21.15.8:6803/1782930059]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-14T13:30:39.768 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:39 smithi008 ceph-mon[24786]: osdmap e7: 1 total, 0 up, 1 in 2024-04-14T13:30:39.768 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:30:39.768 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:39 smithi008 ceph-mon[24786]: from='osd.0 [v2:172.21.15.8:6802/1782930059,v1:172.21.15.8:6803/1782930059]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:30:40.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:39 smithi012 ceph-mon[27236]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:40.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:39 smithi012 ceph-mon[27236]: from='osd.0 [v2:172.21.15.8:6802/1782930059,v1:172.21.15.8:6803/1782930059]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-14T13:30:40.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:39 smithi012 ceph-mon[27236]: osdmap e7: 1 total, 0 up, 1 in 2024-04-14T13:30:40.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:39 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:30:40.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:39 smithi012 ceph-mon[27236]: from='osd.0 [v2:172.21.15.8:6802/1782930059,v1:172.21.15.8:6803/1782930059]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:30:40.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:40 smithi008 ceph-mon[24786]: from='osd.0 [v2:172.21.15.8:6802/1782930059,v1:172.21.15.8:6803/1782930059]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-04-14T13:30:40.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:40 smithi008 ceph-mon[24786]: osdmap e8: 1 total, 0 up, 1 in 2024-04-14T13:30:40.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:40 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:30:40.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:40 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:30:40.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:40 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:40.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:40 smithi008 ceph-mon[24786]: Reconfiguring crash.smithi008 (monmap changed)... 2024-04-14T13:30:40.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:40 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-14T13:30:40.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:40 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:40.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:40 smithi008 ceph-mon[24786]: Reconfiguring daemon crash.smithi008 on smithi008 2024-04-14T13:30:41.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:40 smithi012 ceph-mon[27236]: from='osd.0 [v2:172.21.15.8:6802/1782930059,v1:172.21.15.8:6803/1782930059]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-04-14T13:30:41.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:40 smithi012 ceph-mon[27236]: osdmap e8: 1 total, 0 up, 1 in 2024-04-14T13:30:41.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:40 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:30:41.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:40 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:30:41.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:40 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:41.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:40 smithi012 ceph-mon[27236]: Reconfiguring crash.smithi008 (monmap changed)... 2024-04-14T13:30:41.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:40 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-14T13:30:41.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:40 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:41.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:40 smithi012 ceph-mon[27236]: Reconfiguring daemon crash.smithi008 on smithi008 2024-04-14T13:30:41.064 INFO:teuthology.orchestra.run.smithi008.stdout:Created osd(s) 0 on host 'smithi008' 2024-04-14T13:30:41.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:41 smithi008 ceph-mon[24786]: purged_snaps scrub starts 2024-04-14T13:30:41.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:41 smithi008 ceph-mon[24786]: purged_snaps scrub ok 2024-04-14T13:30:41.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:41 smithi008 ceph-mon[24786]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:41.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:41 smithi008 ceph-mon[24786]: osd.0 [v2:172.21.15.8:6802/1782930059,v1:172.21.15.8:6803/1782930059] boot 2024-04-14T13:30:41.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:41 smithi008 ceph-mon[24786]: osdmap e9: 1 total, 1 up, 1 in 2024-04-14T13:30:41.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:41 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:30:41.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:41 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:41.983 DEBUG:teuthology.orchestra.run.smithi008:osd.0> sudo journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.0.service 2024-04-14T13:30:41.986 INFO:tasks.cephadm:Deploying osd.1 on smithi008 with /dev/vg_nvme/lv_3... 2024-04-14T13:30:41.986 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-14T13:30:42.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:41 smithi012 ceph-mon[27236]: purged_snaps scrub starts 2024-04-14T13:30:42.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:41 smithi012 ceph-mon[27236]: purged_snaps scrub ok 2024-04-14T13:30:42.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:41 smithi012 ceph-mon[27236]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:30:42.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:41 smithi012 ceph-mon[27236]: osd.0 [v2:172.21.15.8:6802/1782930059,v1:172.21.15.8:6803/1782930059] boot 2024-04-14T13:30:42.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:41 smithi012 ceph-mon[27236]: osdmap e9: 1 total, 1 up, 1 in 2024-04-14T13:30:42.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:41 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:30:42.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:41 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:42.801 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:42 smithi008 ceph-mon[24786]: osdmap e10: 1 total, 1 up, 1 in 2024-04-14T13:30:43.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:42 smithi012 ceph-mon[27236]: osdmap e10: 1 total, 1 up, 1 in 2024-04-14T13:30:43.750 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:43 smithi008 ceph-mon[24786]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:30:44.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:43 smithi012 ceph-mon[27236]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:30:44.219 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-14T13:30:44.219 INFO:teuthology.orchestra.run.smithi008.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-14T13:30:44.219 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10+0 records in 2024-04-14T13:30:44.220 INFO:teuthology.orchestra.run.smithi008.stderr:10+0 records out 2024-04-14T13:30:44.220 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00829692 s, 1.3 GB/s 2024-04-14T13:30:44.220 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping successful for: 2024-04-14T13:30:44.822 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi008:vg_nvme/lv_3 2024-04-14T13:30:45.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:45 smithi008 ceph-mon[24786]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:30:45.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:45 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:45.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:45 smithi008 ceph-mon[24786]: Reconfiguring grafana.smithi008 (dependencies changed)... 2024-04-14T13:30:45.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:45 smithi008 ceph-mon[24786]: Reconfiguring daemon grafana.smithi008 on smithi008 2024-04-14T13:30:45.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:45 smithi012 ceph-mon[27236]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:30:45.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:45 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:45.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:45 smithi012 ceph-mon[27236]: Reconfiguring grafana.smithi008 (dependencies changed)... 2024-04-14T13:30:45.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:45 smithi012 ceph-mon[27236]: Reconfiguring daemon grafana.smithi008 on smithi008 2024-04-14T13:30:47.653 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:47 smithi008 ceph-mon[24786]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:30:47.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:47 smithi012 ceph-mon[27236]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:30:48.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:48 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:30:48.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:48 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:30:48.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:48 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:48.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:48 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:30:48.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:48 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:30:48.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:48 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:49.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:49 smithi012 ceph-mon[27236]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:30:49.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:49 smithi012 ceph-mon[27236]: from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:30:49.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:49 smithi008 ceph-mon[24786]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:30:49.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:49 smithi008 ceph-mon[24786]: from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:30:51.549 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:51 smithi012 ceph-mon[27236]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:30:51.549 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:51 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:51.549 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:51 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi012", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-14T13:30:51.549 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:51 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:51.627 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:51 smithi008 ceph-mon[24786]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:30:51.627 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:51 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:51.627 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:51 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi012", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-14T13:30:51.627 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:51 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:52.756 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:52 smithi012 ceph-mon[27236]: Reconfiguring crash.smithi012 (monmap changed)... 2024-04-14T13:30:52.756 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:52 smithi012 ceph-mon[27236]: Reconfiguring daemon crash.smithi012 on smithi012 2024-04-14T13:30:52.756 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:52 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/3206651203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64039dfc-a581-4224-82ea-bff2b958dc1c"}]: dispatch 2024-04-14T13:30:52.756 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:52 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/3206651203' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64039dfc-a581-4224-82ea-bff2b958dc1c"}]': finished 2024-04-14T13:30:52.756 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:52 smithi012 ceph-mon[27236]: osdmap e11: 2 total, 1 up, 2 in 2024-04-14T13:30:52.756 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:52 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:30:52.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:52 smithi008 ceph-mon[24786]: Reconfiguring crash.smithi012 (monmap changed)... 2024-04-14T13:30:52.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:52 smithi008 ceph-mon[24786]: Reconfiguring daemon crash.smithi012 on smithi012 2024-04-14T13:30:52.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:52 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/3206651203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64039dfc-a581-4224-82ea-bff2b958dc1c"}]: dispatch 2024-04-14T13:30:52.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:52 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/3206651203' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64039dfc-a581-4224-82ea-bff2b958dc1c"}]': finished 2024-04-14T13:30:52.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:52 smithi008 ceph-mon[24786]: osdmap e11: 2 total, 1 up, 2 in 2024-04-14T13:30:52.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:52 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:30:54.003 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:53 smithi012 ceph-mon[27236]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:30:54.003 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:53 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:54.003 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:53 smithi012 ceph-mon[27236]: Reconfiguring mgr.smithi012.ksvnxm (monmap changed)... 2024-04-14T13:30:54.003 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:53 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi012.ksvnxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:30:54.003 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:53 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:30:54.004 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:53 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:54.004 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:53 smithi012 ceph-mon[27236]: Reconfiguring daemon mgr.smithi012.ksvnxm on smithi012 2024-04-14T13:30:54.004 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:53 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/1923106179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:30:54.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:53 smithi008 ceph-mon[24786]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:30:54.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:53 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:54.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:53 smithi008 ceph-mon[24786]: Reconfiguring mgr.smithi012.ksvnxm (monmap changed)... 2024-04-14T13:30:54.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:53 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi012.ksvnxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:30:54.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:53 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:30:54.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:53 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:54.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:53 smithi008 ceph-mon[24786]: Reconfiguring daemon mgr.smithi012.ksvnxm on smithi012 2024-04-14T13:30:54.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:53 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/1923106179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:30:55.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:55 smithi012 ceph-mon[27236]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:30:55.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:55 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:55.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:55 smithi012 ceph-mon[27236]: Reconfiguring mon.smithi012 (monmap changed)... 2024-04-14T13:30:55.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:55 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:30:55.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:55 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:30:55.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:55 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:55.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:55 smithi012 ceph-mon[27236]: Reconfiguring daemon mon.smithi012 on smithi012 2024-04-14T13:30:55.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:55 smithi008 ceph-mon[24786]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:30:55.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:55 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:55.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:55 smithi008 ceph-mon[24786]: Reconfiguring mon.smithi012 (monmap changed)... 2024-04-14T13:30:55.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:55 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:30:55.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:55 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:30:55.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:55 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:55.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:55 smithi008 ceph-mon[24786]: Reconfiguring daemon mon.smithi012 on smithi012 2024-04-14T13:30:56.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:56 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:56.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:56 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T13:30:56.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:56 smithi012 ceph-mon[27236]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T13:30:56.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:56 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:30:56.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:56 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T13:30:56.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:56 smithi008 ceph-mon[24786]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T13:30:57.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:57 smithi012 ceph-mon[27236]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:30:57.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:57 smithi008 ceph-mon[24786]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:30:58.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:58 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T13:30:58.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:58 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:58.849 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:58 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T13:30:58.849 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:58 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:30:59.745 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:59 smithi008 ceph-mon[24786]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:30:59.745 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:30:59 smithi008 ceph-mon[24786]: Deploying daemon osd.1 on smithi008 2024-04-14T13:30:59.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:59 smithi012 ceph-mon[27236]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:30:59.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:30:59 smithi012 ceph-mon[27236]: Deploying daemon osd.1 on smithi008 2024-04-14T13:31:01.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:01 smithi008 ceph-mon[24786]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:31:01.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:01 smithi012 ceph-mon[27236]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:31:03.565 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:03 smithi008 ceph-mon[24786]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:31:03.566 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:03 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:03.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:03 smithi012 ceph-mon[27236]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:31:03.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:03 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:05.490 INFO:teuthology.orchestra.run.smithi008.stdout:Created osd(s) 1 on host 'smithi008' 2024-04-14T13:31:05.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:05 smithi012 ceph-mon[27236]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:31:05.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:05 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:05.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:05 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.8:9093"}]: dispatch 2024-04-14T13:31:05.846 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:05 smithi008 ceph-mon[24786]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:31:05.846 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:05 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:05.847 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:05 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.8:9093"}]: dispatch 2024-04-14T13:31:06.137 DEBUG:teuthology.orchestra.run.smithi008:osd.1> sudo journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.1.service 2024-04-14T13:31:06.139 INFO:tasks.cephadm:Deploying osd.2 on smithi008 with /dev/vg_nvme/lv_2... 2024-04-14T13:31:06.140 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-14T13:31:06.500 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:06 smithi012 ceph-mon[27236]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.8:9093"}]: dispatch 2024-04-14T13:31:06.771 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:06 smithi008 ceph-mon[24786]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.8:9093"}]: dispatch 2024-04-14T13:31:06.771 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:06.771 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T13:31:06.771 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:06 smithi008 ceph-mon[24786]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T13:31:06.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.8:3000"}]: dispatch 2024-04-14T13:31:06.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:06 smithi008 ceph-mon[24786]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.8:3000"}]: dispatch 2024-04-14T13:31:06.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:06.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:31:06.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:06 smithi008 ceph-mon[24786]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:31:06.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.8:9095"}]: dispatch 2024-04-14T13:31:06.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:06 smithi008 ceph-mon[24786]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.8:9095"}]: dispatch 2024-04-14T13:31:06.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:06.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:31:06.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:31:06.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:31:06.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:06 smithi008 ceph-mon[24786]: from='osd.1 [v2:172.21.15.8:6810/2761200318,v1:172.21.15.8:6811/2761200318]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T13:31:06.797 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:06.797 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T13:31:06.797 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:06 smithi012 ceph-mon[27236]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T13:31:06.797 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.8:3000"}]: dispatch 2024-04-14T13:31:06.797 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:06 smithi012 ceph-mon[27236]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.8:3000"}]: dispatch 2024-04-14T13:31:06.798 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:06.798 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:31:06.798 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:06 smithi012 ceph-mon[27236]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:31:06.798 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.8:9095"}]: dispatch 2024-04-14T13:31:06.798 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:06 smithi012 ceph-mon[27236]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.8:9095"}]: dispatch 2024-04-14T13:31:06.798 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:06.798 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:31:06.798 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:31:06.798 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:31:06.798 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:06 smithi012 ceph-mon[27236]: from='osd.1 [v2:172.21.15.8:6810/2761200318,v1:172.21.15.8:6811/2761200318]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T13:31:07.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:07 smithi012 ceph-mon[27236]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:31:07.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:07 smithi012 ceph-mon[27236]: from='osd.1 [v2:172.21.15.8:6810/2761200318,v1:172.21.15.8:6811/2761200318]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-14T13:31:07.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:07 smithi012 ceph-mon[27236]: osdmap e12: 2 total, 1 up, 2 in 2024-04-14T13:31:07.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:07 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:31:07.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:07 smithi012 ceph-mon[27236]: from='osd.1 [v2:172.21.15.8:6810/2761200318,v1:172.21.15.8:6811/2761200318]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:31:07.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:07 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:07.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:07 smithi008 ceph-mon[24786]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:31:07.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:07 smithi008 ceph-mon[24786]: from='osd.1 [v2:172.21.15.8:6810/2761200318,v1:172.21.15.8:6811/2761200318]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-14T13:31:07.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:07 smithi008 ceph-mon[24786]: osdmap e12: 2 total, 1 up, 2 in 2024-04-14T13:31:07.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:07 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:31:07.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:07 smithi008 ceph-mon[24786]: from='osd.1 [v2:172.21.15.8:6810/2761200318,v1:172.21.15.8:6811/2761200318]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:31:07.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:07 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:07.852 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:31:07 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1[38101]: 2024-04-14T13:31:07.530+0000 7f7a48d8e700 -1 osd.1 0 waiting for initial osdmap 2024-04-14T13:31:07.852 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:31:07 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1[38101]: 2024-04-14T13:31:07.533+0000 7f7a4071f700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:31:08.656 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-14T13:31:08.656 INFO:teuthology.orchestra.run.smithi008.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-14T13:31:08.656 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10+0 records in 2024-04-14T13:31:08.656 INFO:teuthology.orchestra.run.smithi008.stderr:10+0 records out 2024-04-14T13:31:08.656 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00963372 s, 1.1 GB/s 2024-04-14T13:31:08.656 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping successful for: 2024-04-14T13:31:08.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:08 smithi012 ceph-mon[27236]: from='osd.1 [v2:172.21.15.8:6810/2761200318,v1:172.21.15.8:6811/2761200318]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-04-14T13:31:08.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:08 smithi012 ceph-mon[27236]: osdmap e13: 2 total, 1 up, 2 in 2024-04-14T13:31:08.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:08 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:31:08.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:08 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:31:08.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:08 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:31:08.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:08 smithi008 ceph-mon[24786]: from='osd.1 [v2:172.21.15.8:6810/2761200318,v1:172.21.15.8:6811/2761200318]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-04-14T13:31:08.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:08 smithi008 ceph-mon[24786]: osdmap e13: 2 total, 1 up, 2 in 2024-04-14T13:31:08.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:08 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:31:08.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:08 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:31:08.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:08 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:31:09.364 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi008:vg_nvme/lv_2 2024-04-14T13:31:09.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:09 smithi008 ceph-mon[24786]: purged_snaps scrub starts 2024-04-14T13:31:09.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:09 smithi008 ceph-mon[24786]: purged_snaps scrub ok 2024-04-14T13:31:09.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:09 smithi008 ceph-mon[24786]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:31:09.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:09 smithi008 ceph-mon[24786]: osd.1 [v2:172.21.15.8:6810/2761200318,v1:172.21.15.8:6811/2761200318] boot 2024-04-14T13:31:09.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:09 smithi008 ceph-mon[24786]: osdmap e14: 2 total, 2 up, 2 in 2024-04-14T13:31:09.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:09 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:31:09.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:09 smithi012 ceph-mon[27236]: purged_snaps scrub starts 2024-04-14T13:31:09.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:09 smithi012 ceph-mon[27236]: purged_snaps scrub ok 2024-04-14T13:31:09.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:09 smithi012 ceph-mon[27236]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:31:09.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:09 smithi012 ceph-mon[27236]: osd.1 [v2:172.21.15.8:6810/2761200318,v1:172.21.15.8:6811/2761200318] boot 2024-04-14T13:31:09.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:09 smithi012 ceph-mon[27236]: osdmap e14: 2 total, 2 up, 2 in 2024-04-14T13:31:09.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:09 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:31:10.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:10 smithi012 ceph-mon[27236]: osdmap e15: 2 total, 2 up, 2 in 2024-04-14T13:31:10.830 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:10 smithi008 ceph-mon[24786]: osdmap e15: 2 total, 2 up, 2 in 2024-04-14T13:31:11.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:11 smithi012 ceph-mon[27236]: pgmap v52: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:11.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:11 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:31:11.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:11 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:31:11.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:11 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:31:11.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:11 smithi008 ceph-mon[24786]: pgmap v52: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:11.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:11 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:31:11.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:11 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:31:11.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:11 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:31:12.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:12 smithi008 ceph-mon[24786]: from='client.14276 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:31:13.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:12 smithi012 ceph-mon[27236]: from='client.14276 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:31:13.707 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:13 smithi008 ceph-mon[24786]: pgmap v53: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:14.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:13 smithi012 ceph-mon[27236]: pgmap v53: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:15.762 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:15 smithi008 ceph-mon[24786]: pgmap v54: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:15.762 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:15 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/3001534892' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee8d332d-b536-4dfe-b0a2-f43d79149677"}]: dispatch 2024-04-14T13:31:15.763 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:15 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/3001534892' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee8d332d-b536-4dfe-b0a2-f43d79149677"}]': finished 2024-04-14T13:31:15.763 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:15 smithi008 ceph-mon[24786]: osdmap e16: 3 total, 2 up, 3 in 2024-04-14T13:31:15.763 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:15 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:31:15.763 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:15 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:15.763 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:15 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/3605075052' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:31:15.851 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:15 smithi012 ceph-mon[27236]: pgmap v54: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:15.851 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:15 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/3001534892' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee8d332d-b536-4dfe-b0a2-f43d79149677"}]: dispatch 2024-04-14T13:31:15.852 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:15 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/3001534892' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee8d332d-b536-4dfe-b0a2-f43d79149677"}]': finished 2024-04-14T13:31:15.852 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:15 smithi012 ceph-mon[27236]: osdmap e16: 3 total, 2 up, 3 in 2024-04-14T13:31:15.852 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:15 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:31:15.852 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:15 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:15.852 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:15 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/3605075052' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:31:17.812 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:17 smithi008 ceph-mon[24786]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:18.037 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:17 smithi012 ceph-mon[27236]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:19.781 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:19 smithi008 ceph-mon[24786]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:19.873 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:19 smithi012 ceph-mon[27236]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:21.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:20 smithi008 ceph-mon[24786]: Detected new or changed devices on smithi008 2024-04-14T13:31:21.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:20 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:21.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:20 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:31:21.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:20 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:31:21.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:20 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:21.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:20 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:21.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:20 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:31:21.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:20 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:31:21.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:20 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:31:21.169 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:20 smithi012 ceph-mon[27236]: Detected new or changed devices on smithi008 2024-04-14T13:31:21.169 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:20 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:21.169 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:20 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:31:21.169 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:20 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:31:21.169 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:20 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:21.169 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:20 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:21.169 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:20 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:31:21.169 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:20 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:31:21.169 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:20 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:31:22.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:21 smithi012 ceph-mon[27236]: Adjusting osd_memory_target on smithi008 to 6434M 2024-04-14T13:31:22.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:21 smithi012 ceph-mon[27236]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:22.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:21 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T13:31:22.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:21 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:31:22.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:21 smithi012 ceph-mon[27236]: Deploying daemon osd.2 on smithi008 2024-04-14T13:31:22.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:21 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:22.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:21 smithi008 ceph-mon[24786]: Adjusting osd_memory_target on smithi008 to 6434M 2024-04-14T13:31:22.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:21 smithi008 ceph-mon[24786]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:22.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:21 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T13:31:22.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:21 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:31:22.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:21 smithi008 ceph-mon[24786]: Deploying daemon osd.2 on smithi008 2024-04-14T13:31:22.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:21 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:23.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:22 smithi008 ceph-mon[24786]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:23.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:22 smithi012 ceph-mon[27236]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:25.298 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:24 smithi008 ceph-mon[24786]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:25.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:24 smithi012 ceph-mon[27236]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:27.547 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:27 smithi008 ceph-mon[24786]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:27.547 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:27 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:27.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:27 smithi012 ceph-mon[27236]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:27.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:27 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:29.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:29 smithi008 ceph-mon[24786]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:29.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:29 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:30.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:29 smithi012 ceph-mon[27236]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:30.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:29 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:32.018 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:31 smithi008 ceph-mon[24786]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:32.018 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:31 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:32.019 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:31 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:32.019 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:31 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:31:32.019 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:31 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:31:32.019 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:31 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:31:32.019 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:31 smithi008 ceph-mon[24786]: from='osd.2 [v2:172.21.15.8:6818/3582626304,v1:172.21.15.8:6819/3582626304]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T13:31:32.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:31 smithi012 ceph-mon[27236]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:32.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:31 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:32.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:31 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:32.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:31 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:31:32.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:31 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:31:32.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:31 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:31:32.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:31 smithi012 ceph-mon[27236]: from='osd.2 [v2:172.21.15.8:6818/3582626304,v1:172.21.15.8:6819/3582626304]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T13:31:32.541 INFO:teuthology.orchestra.run.smithi008.stdout:Created osd(s) 2 on host 'smithi008' 2024-04-14T13:31:32.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:32 smithi012 ceph-mon[27236]: from='osd.2 [v2:172.21.15.8:6818/3582626304,v1:172.21.15.8:6819/3582626304]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-14T13:31:32.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:32 smithi012 ceph-mon[27236]: osdmap e17: 3 total, 2 up, 3 in 2024-04-14T13:31:32.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:31:32.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:32 smithi012 ceph-mon[27236]: from='osd.2 [v2:172.21.15.8:6818/3582626304,v1:172.21.15.8:6819/3582626304]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:31:32.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:31:32.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:31:32.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:32.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:32.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:32 smithi008 ceph-mon[24786]: from='osd.2 [v2:172.21.15.8:6818/3582626304,v1:172.21.15.8:6819/3582626304]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-14T13:31:32.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:32 smithi008 ceph-mon[24786]: osdmap e17: 3 total, 2 up, 3 in 2024-04-14T13:31:32.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:31:32.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:32 smithi008 ceph-mon[24786]: from='osd.2 [v2:172.21.15.8:6818/3582626304,v1:172.21.15.8:6819/3582626304]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:31:32.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:31:32.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:31:32.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:32.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:33.497 DEBUG:teuthology.orchestra.run.smithi008:osd.2> sudo journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.2.service 2024-04-14T13:31:33.499 INFO:tasks.cephadm:Deploying osd.3 on smithi008 with /dev/vg_nvme/lv_1... 2024-04-14T13:31:33.500 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-14T13:31:34.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:33 smithi012 ceph-mon[27236]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:34.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:33 smithi012 ceph-mon[27236]: from='osd.2 [v2:172.21.15.8:6818/3582626304,v1:172.21.15.8:6819/3582626304]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-04-14T13:31:34.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:33 smithi012 ceph-mon[27236]: osdmap e18: 3 total, 2 up, 3 in 2024-04-14T13:31:34.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:33 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:31:34.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:33 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:31:34.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:33 smithi008 ceph-mon[24786]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:34.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:33 smithi008 ceph-mon[24786]: from='osd.2 [v2:172.21.15.8:6818/3582626304,v1:172.21.15.8:6819/3582626304]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-04-14T13:31:34.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:33 smithi008 ceph-mon[24786]: osdmap e18: 3 total, 2 up, 3 in 2024-04-14T13:31:34.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:33 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:31:34.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:33 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:31:34.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:34 smithi008 ceph-mon[24786]: purged_snaps scrub starts 2024-04-14T13:31:34.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:34 smithi008 ceph-mon[24786]: purged_snaps scrub ok 2024-04-14T13:31:34.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:34 smithi008 ceph-mon[24786]: osd.2 [v2:172.21.15.8:6818/3582626304,v1:172.21.15.8:6819/3582626304] boot 2024-04-14T13:31:34.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:34 smithi008 ceph-mon[24786]: osdmap e19: 3 total, 3 up, 3 in 2024-04-14T13:31:34.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:34 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:31:34.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:34 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-14T13:31:35.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:34 smithi012 ceph-mon[27236]: purged_snaps scrub starts 2024-04-14T13:31:35.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:34 smithi012 ceph-mon[27236]: purged_snaps scrub ok 2024-04-14T13:31:35.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:34 smithi012 ceph-mon[27236]: osd.2 [v2:172.21.15.8:6818/3582626304,v1:172.21.15.8:6819/3582626304] boot 2024-04-14T13:31:35.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:34 smithi012 ceph-mon[27236]: osdmap e19: 3 total, 3 up, 3 in 2024-04-14T13:31:35.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:34 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:31:35.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:34 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-14T13:31:35.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:35 smithi008 ceph-mon[24786]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:35.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:35 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-14T13:31:35.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:35 smithi008 ceph-mon[24786]: osdmap e20: 3 total, 3 up, 3 in 2024-04-14T13:31:35.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:35 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-14T13:31:35.964 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-14T13:31:35.965 INFO:teuthology.orchestra.run.smithi008.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-14T13:31:35.965 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10+0 records in 2024-04-14T13:31:35.965 INFO:teuthology.orchestra.run.smithi008.stderr:10+0 records out 2024-04-14T13:31:35.965 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00855155 s, 1.2 GB/s 2024-04-14T13:31:35.965 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping successful for: 2024-04-14T13:31:36.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:35 smithi012 ceph-mon[27236]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:31:36.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:35 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-14T13:31:36.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:35 smithi012 ceph-mon[27236]: osdmap e20: 3 total, 3 up, 3 in 2024-04-14T13:31:36.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:35 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-14T13:31:36.607 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi008:vg_nvme/lv_1 2024-04-14T13:31:36.957 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:36 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-14T13:31:36.957 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:36 smithi008 ceph-mon[24786]: osdmap e21: 3 total, 3 up, 3 in 2024-04-14T13:31:37.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:36 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-14T13:31:37.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:36 smithi012 ceph-mon[27236]: osdmap e21: 3 total, 3 up, 3 in 2024-04-14T13:31:37.224 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:31:36 smithi008 sudo[43268]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T13:31:37.224 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:31:36 smithi008 sudo[43268]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:31:37.224 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:31:36 smithi008 sudo[43268]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:31:37.504 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:31:37 smithi008 sudo[43290]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T13:31:37.505 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:31:37 smithi008 sudo[43290]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:31:37.505 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:31:37 smithi008 sudo[43290]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:31:37.774 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:31:37 smithi008 sudo[43313]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T13:31:37.774 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:31:37 smithi008 sudo[43313]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:31:37.775 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:31:37 smithi008 sudo[43313]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:31:38.033 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:31:37 smithi008 sudo[43331]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T13:31:38.033 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:31:37 smithi008 sudo[43331]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:31:38.033 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:31:37 smithi008 sudo[43331]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:31:38.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:37 smithi008 ceph-mon[24786]: pgmap v71: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:38.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:37 smithi008 ceph-mon[24786]: osdmap e22: 3 total, 3 up, 3 in 2024-04-14T13:31:38.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:37 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:38.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:37 smithi012 ceph-mon[27236]: pgmap v71: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:38.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:37 smithi012 ceph-mon[27236]: osdmap e22: 3 total, 3 up, 3 in 2024-04-14T13:31:38.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:37 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:38.305 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:31:38 smithi008 sudo[43355]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T13:31:38.305 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:31:38 smithi008 sudo[43355]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:31:38.305 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:31:38 smithi008 sudo[43355]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:31:38.561 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:31:38 smithi008 sudo[43386]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T13:31:38.561 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:31:38 smithi008 sudo[43386]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:31:38.561 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:31:38 smithi008 sudo[43386]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:31:38.827 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:38 smithi008 sudo[43389]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T13:31:38.828 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:38 smithi008 sudo[43389]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:31:38.828 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:38 smithi008 sudo[43389]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:31:39.084 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:38 smithi008 sudo[43411]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T13:31:39.084 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:38 smithi008 sudo[43411]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:31:39.084 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:38 smithi008 sudo[43411]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:31:39.357 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:39 smithi012 sudo[29552]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T13:31:39.357 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:39 smithi012 sudo[29552]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:31:39.357 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:39 smithi012 sudo[29552]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:31:39.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:39 smithi012 sudo[29555]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T13:31:39.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:39 smithi012 sudo[29555]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:31:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:39 smithi012 sudo[29555]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:31:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:39 smithi012 ceph-mon[27236]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:39 smithi012 ceph-mon[27236]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T13:31:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:39 smithi012 ceph-mon[27236]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T13:31:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:39 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:31:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:39 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:31:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:39 smithi012 ceph-mon[27236]: mgrmap e19: smithi008.yemlhr(active, since 2m), standbys: smithi012.ksvnxm 2024-04-14T13:31:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:39 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:31:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:39 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:31:39.840 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:39 smithi008 ceph-mon[24786]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:39.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:39 smithi008 ceph-mon[24786]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T13:31:39.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:39 smithi008 ceph-mon[24786]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T13:31:39.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:31:39.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:31:39.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:39 smithi008 ceph-mon[24786]: mgrmap e19: smithi008.yemlhr(active, since 2m), standbys: smithi012.ksvnxm 2024-04-14T13:31:39.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:31:39.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:31:40.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:40 smithi008 ceph-mon[24786]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T13:31:40.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:40 smithi008 ceph-mon[24786]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T13:31:40.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:40 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:40.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:40 smithi008 ceph-mon[24786]: from='client.14300 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:31:40.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:40 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:31:40.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:40 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:31:40.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:40 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:31:40.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:40 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:40.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:40 smithi012 ceph-mon[27236]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T13:31:40.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:40 smithi012 ceph-mon[27236]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T13:31:40.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:40 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:40.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:40 smithi012 ceph-mon[27236]: from='client.14300 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:31:40.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:40 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:31:40.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:40 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:31:40.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:40 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:31:40.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:40 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:41.676 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:41 smithi008 ceph-mon[24786]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:41.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:41 smithi012 ceph-mon[27236]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:42.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:42 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/2614300291' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65bead61-5127-4144-8704-41a61d19aac0"}]: dispatch 2024-04-14T13:31:42.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:42 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/2614300291' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "65bead61-5127-4144-8704-41a61d19aac0"}]': finished 2024-04-14T13:31:42.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:42 smithi012 ceph-mon[27236]: osdmap e23: 4 total, 3 up, 4 in 2024-04-14T13:31:42.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:42 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:31:42.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:42 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2614300291' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65bead61-5127-4144-8704-41a61d19aac0"}]: dispatch 2024-04-14T13:31:42.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:42 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2614300291' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "65bead61-5127-4144-8704-41a61d19aac0"}]': finished 2024-04-14T13:31:42.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:42 smithi008 ceph-mon[24786]: osdmap e23: 4 total, 3 up, 4 in 2024-04-14T13:31:42.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:42 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:31:43.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:43 smithi012 ceph-mon[27236]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:43.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:43 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/4258475938' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:31:43.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:43 smithi008 ceph-mon[24786]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:43.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:43 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/4258475938' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:31:45.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:45 smithi008 ceph-mon[24786]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:45.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:45 smithi012 ceph-mon[27236]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:47.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:47 smithi008 ceph-mon[24786]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:47.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:47 smithi012 ceph-mon[27236]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:48.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:48 smithi008 ceph-mon[24786]: Detected new or changed devices on smithi008 2024-04-14T13:31:48.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:48 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:48.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:48 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:31:48.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:48 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:31:48.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:48 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:31:48.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:48 smithi008 ceph-mon[24786]: Adjusting osd_memory_target on smithi008 to 4289M 2024-04-14T13:31:48.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:48 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:48.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:48 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:48.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:48 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:31:48.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:48 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:31:48.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:48 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:31:48.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:48 smithi008 ceph-mon[24786]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:48 smithi012 ceph-mon[27236]: Detected new or changed devices on smithi008 2024-04-14T13:31:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:48 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:48 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:31:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:48 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:31:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:48 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:31:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:48 smithi012 ceph-mon[27236]: Adjusting osd_memory_target on smithi008 to 4289M 2024-04-14T13:31:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:48 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:48 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:48 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:31:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:48 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:31:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:48 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:31:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:48 smithi012 ceph-mon[27236]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:50.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:49 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T13:31:50.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:49 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:31:50.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:49 smithi008 ceph-mon[24786]: Deploying daemon osd.3 on smithi008 2024-04-14T13:31:50.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:49 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T13:31:50.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:49 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:31:50.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:49 smithi012 ceph-mon[27236]: Deploying daemon osd.3 on smithi008 2024-04-14T13:31:51.649 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:51 smithi008 ceph-mon[24786]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:51.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:51 smithi012 ceph-mon[27236]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:53.655 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:53 smithi008 ceph-mon[24786]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:53.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:53 smithi012 ceph-mon[27236]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:55.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:55 smithi008 ceph-mon[24786]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:55.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:55 smithi012 ceph-mon[27236]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:57.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:56 smithi008 ceph-mon[24786]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:57.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:56 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:57.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:56 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:57.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:56 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:57.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:56 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:31:57.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:56 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:31:57.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:56 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:31:57.205 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:56 smithi012 ceph-mon[27236]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:57.205 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:56 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:57.205 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:56 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:57.205 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:56 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:31:57.206 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:56 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:31:57.206 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:56 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:31:57.206 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:56 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:31:58.946 INFO:teuthology.orchestra.run.smithi008.stdout:Created osd(s) 3 on host 'smithi008' 2024-04-14T13:31:59.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:58 smithi008 ceph-mon[24786]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:59.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:58 smithi012 ceph-mon[27236]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:31:59.644 DEBUG:teuthology.orchestra.run.smithi008:osd.3> sudo journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.3.service 2024-04-14T13:31:59.646 INFO:tasks.cephadm:Deploying osd.4 on smithi012 with /dev/vg_nvme/lv_4... 2024-04-14T13:31:59.647 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-14T13:32:00.265 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:59 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:00.265 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:31:59 smithi012 ceph-mon[27236]: from='osd.3 [v2:172.21.15.8:6826/2354457054,v1:172.21.15.8:6827/2354457054]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T13:32:00.306 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:59 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:00.307 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:31:59 smithi008 ceph-mon[24786]: from='osd.3 [v2:172.21.15.8:6826/2354457054,v1:172.21.15.8:6827/2354457054]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T13:32:01.209 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:00 smithi008 ceph-mon[24786]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:32:01.209 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:00 smithi008 ceph-mon[24786]: from='osd.3 [v2:172.21.15.8:6826/2354457054,v1:172.21.15.8:6827/2354457054]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-14T13:32:01.209 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:00 smithi008 ceph-mon[24786]: osdmap e24: 4 total, 3 up, 4 in 2024-04-14T13:32:01.209 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:00 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:32:01.210 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:00 smithi008 ceph-mon[24786]: from='osd.3 [v2:172.21.15.8:6826/2354457054,v1:172.21.15.8:6827/2354457054]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:32:01.210 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:32:00 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3[45822]: 2024-04-14T13:32:00.952+0000 7fddc5228700 -1 osd.3 0 waiting for initial osdmap 2024-04-14T13:32:01.210 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:32:00 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3[45822]: 2024-04-14T13:32:00.956+0000 7fddbfbbf700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:32:01.237 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:00 smithi012 ceph-mon[27236]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:32:01.237 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:00 smithi012 ceph-mon[27236]: from='osd.3 [v2:172.21.15.8:6826/2354457054,v1:172.21.15.8:6827/2354457054]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-14T13:32:01.238 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:00 smithi012 ceph-mon[27236]: osdmap e24: 4 total, 3 up, 4 in 2024-04-14T13:32:01.238 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:00 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:32:01.238 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:00 smithi012 ceph-mon[27236]: from='osd.3 [v2:172.21.15.8:6826/2354457054,v1:172.21.15.8:6827/2354457054]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:32:01.767 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-14T13:32:01.767 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-14T13:32:01.767 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-04-14T13:32:01.767 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-04-14T13:32:01.767 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01139 s, 921 MB/s 2024-04-14T13:32:01.767 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-04-14T13:32:02.007 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:01 smithi008 ceph-mon[24786]: from='osd.3 [v2:172.21.15.8:6826/2354457054,v1:172.21.15.8:6827/2354457054]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-04-14T13:32:02.007 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:01 smithi008 ceph-mon[24786]: osdmap e25: 4 total, 3 up, 4 in 2024-04-14T13:32:02.008 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:01 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:32:02.008 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:01 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:32:02.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:01 smithi012 ceph-mon[27236]: from='osd.3 [v2:172.21.15.8:6826/2354457054,v1:172.21.15.8:6827/2354457054]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-04-14T13:32:02.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:01 smithi012 ceph-mon[27236]: osdmap e25: 4 total, 3 up, 4 in 2024-04-14T13:32:02.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:01 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:32:02.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:01 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:32:02.390 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi012:vg_nvme/lv_4 2024-04-14T13:32:03.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:02 smithi012 ceph-mon[27236]: purged_snaps scrub starts 2024-04-14T13:32:03.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:02 smithi012 ceph-mon[27236]: purged_snaps scrub ok 2024-04-14T13:32:03.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:02 smithi012 ceph-mon[27236]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:32:03.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:02 smithi012 ceph-mon[27236]: osd.3 [v2:172.21.15.8:6826/2354457054,v1:172.21.15.8:6827/2354457054] boot 2024-04-14T13:32:03.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:02 smithi012 ceph-mon[27236]: osdmap e26: 4 total, 4 up, 4 in 2024-04-14T13:32:03.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:02 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:32:03.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:02 smithi008 ceph-mon[24786]: purged_snaps scrub starts 2024-04-14T13:32:03.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:02 smithi008 ceph-mon[24786]: purged_snaps scrub ok 2024-04-14T13:32:03.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:02 smithi008 ceph-mon[24786]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:32:03.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:02 smithi008 ceph-mon[24786]: osd.3 [v2:172.21.15.8:6826/2354457054,v1:172.21.15.8:6827/2354457054] boot 2024-04-14T13:32:03.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:02 smithi008 ceph-mon[24786]: osdmap e26: 4 total, 4 up, 4 in 2024-04-14T13:32:03.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:02 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:32:04.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:03 smithi008 ceph-mon[24786]: osdmap e27: 4 total, 4 up, 4 in 2024-04-14T13:32:04.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:03 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:04.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:03 smithi012 ceph-mon[27236]: osdmap e27: 4 total, 4 up, 4 in 2024-04-14T13:32:04.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:03 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:05.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:05 smithi012 ceph-mon[27236]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:05.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:05 smithi012 ceph-mon[27236]: from='client.24153 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:32:05.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:05 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:32:05.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:05 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:32:05.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:05 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:05.349 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:05 smithi008 ceph-mon[24786]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:05.349 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:05 smithi008 ceph-mon[24786]: from='client.24153 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:32:05.350 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:05 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:32:05.350 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:05 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:32:05.350 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:05 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:07.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:07 smithi012 ceph-mon[27236]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:07.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:07 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:07.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:07 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:32:07.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:07 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:32:07.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:07 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:32:07.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:07 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:32:07.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:07 smithi012 ceph-mon[27236]: Adjusting osd_memory_target on smithi008 to 3217M 2024-04-14T13:32:07.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:07 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:07.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:07 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:07.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:07 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:32:07.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:07 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:07.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:07 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:32:07.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:07 smithi012 ceph-mon[27236]: from='client.? 172.21.15.12:0/2957853418' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e81bc030-196f-4623-9d12-2bc124c12c1b"}]: dispatch 2024-04-14T13:32:07.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:07 smithi012 ceph-mon[27236]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e81bc030-196f-4623-9d12-2bc124c12c1b"}]: dispatch 2024-04-14T13:32:07.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:07 smithi012 ceph-mon[27236]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e81bc030-196f-4623-9d12-2bc124c12c1b"}]': finished 2024-04-14T13:32:07.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:07 smithi012 ceph-mon[27236]: osdmap e28: 5 total, 4 up, 5 in 2024-04-14T13:32:07.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:07 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:32:07.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:07 smithi008 ceph-mon[24786]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:07.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:07 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:07.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:07 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:32:07.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:07 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:32:07.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:07 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:32:07.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:07 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:32:07.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:07 smithi008 ceph-mon[24786]: Adjusting osd_memory_target on smithi008 to 3217M 2024-04-14T13:32:07.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:07 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:07.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:07 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:07.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:07 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:32:07.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:07 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:07.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:07 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:32:07.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:07 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/2957853418' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e81bc030-196f-4623-9d12-2bc124c12c1b"}]: dispatch 2024-04-14T13:32:07.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:07 smithi008 ceph-mon[24786]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e81bc030-196f-4623-9d12-2bc124c12c1b"}]: dispatch 2024-04-14T13:32:07.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:07 smithi008 ceph-mon[24786]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e81bc030-196f-4623-9d12-2bc124c12c1b"}]': finished 2024-04-14T13:32:07.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:07 smithi008 ceph-mon[24786]: osdmap e28: 5 total, 4 up, 5 in 2024-04-14T13:32:07.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:07 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:32:08.290 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:08 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/3238355783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:32:08.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:08 smithi012 ceph-mon[27236]: from='client.? 172.21.15.12:0/3238355783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:32:09.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:09 smithi012 ceph-mon[27236]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:09.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:09 smithi008 ceph-mon[24786]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:11.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:10 smithi012 ceph-mon[27236]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:11.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:10 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:11.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:10 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:11.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:10 smithi008 ceph-mon[24786]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:11.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:10 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:11.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:10 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:13.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:13 smithi012 ceph-mon[27236]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:13.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:13 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T13:32:13.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:13 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:13.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:13 smithi012 ceph-mon[27236]: Deploying daemon osd.4 on smithi012 2024-04-14T13:32:13.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:13 smithi008 ceph-mon[24786]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:13.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:13 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T13:32:13.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:13 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:13.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:13 smithi008 ceph-mon[24786]: Deploying daemon osd.4 on smithi012 2024-04-14T13:32:15.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:15 smithi012 ceph-mon[27236]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:15.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:15 smithi008 ceph-mon[24786]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:17.674 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:17 smithi012 ceph-mon[27236]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:17.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:17 smithi008 ceph-mon[24786]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:18.950 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:18 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:18.951 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:18 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:32:18.951 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:18 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:18.951 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:18 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:32:19.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:18 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:19.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:18 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:32:19.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:18 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:19.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:18 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:32:20.044 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:19 smithi008 ceph-mon[24786]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:20.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:19 smithi012 ceph-mon[27236]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:20.757 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 4 on host 'smithi012' 2024-04-14T13:32:21.443 DEBUG:teuthology.orchestra.run.smithi012:osd.4> sudo journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.4.service 2024-04-14T13:32:21.445 INFO:tasks.cephadm:Deploying osd.5 on smithi012 with /dev/vg_nvme/lv_3... 2024-04-14T13:32:21.446 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-14T13:32:21.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:21 smithi012 ceph-mon[27236]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:21.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:21 smithi012 ceph-mon[27236]: from='osd.4 [v2:172.21.15.12:6800/2473086748,v1:172.21.15.12:6801/2473086748]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T13:32:21.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:21 smithi012 ceph-mon[27236]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T13:32:21.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:21 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:22.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:21 smithi008 ceph-mon[24786]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:22.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:21 smithi008 ceph-mon[24786]: from='osd.4 [v2:172.21.15.12:6800/2473086748,v1:172.21.15.12:6801/2473086748]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T13:32:22.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:21 smithi008 ceph-mon[24786]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T13:32:22.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:21 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:23.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:22 smithi012 ceph-mon[27236]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-14T13:32:23.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:22 smithi012 ceph-mon[27236]: osdmap e29: 5 total, 4 up, 5 in 2024-04-14T13:32:23.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:22 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:32:23.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:22 smithi012 ceph-mon[27236]: from='osd.4 [v2:172.21.15.12:6800/2473086748,v1:172.21.15.12:6801/2473086748]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:32:23.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:22 smithi012 ceph-mon[27236]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:32:23.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:22 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:23.051 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:32:22 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[31414]: 2024-04-14T13:32:22.665+0000 7f891e961700 -1 osd.4 0 waiting for initial osdmap 2024-04-14T13:32:23.051 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:32:22 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[31414]: 2024-04-14T13:32:22.670+0000 7f891b2fc700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:32:23.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:22 smithi008 ceph-mon[24786]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-14T13:32:23.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:22 smithi008 ceph-mon[24786]: osdmap e29: 5 total, 4 up, 5 in 2024-04-14T13:32:23.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:22 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:32:23.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:22 smithi008 ceph-mon[24786]: from='osd.4 [v2:172.21.15.12:6800/2473086748,v1:172.21.15.12:6801/2473086748]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:32:23.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:22 smithi008 ceph-mon[24786]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:32:23.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:22 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:24.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:23 smithi012 ceph-mon[27236]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:24.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:23 smithi012 ceph-mon[27236]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-14T13:32:24.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:23 smithi012 ceph-mon[27236]: osdmap e30: 5 total, 4 up, 5 in 2024-04-14T13:32:24.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:23 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:32:24.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:23 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:32:24.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:23 smithi008 ceph-mon[24786]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:24.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:23 smithi008 ceph-mon[24786]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-14T13:32:24.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:23 smithi008 ceph-mon[24786]: osdmap e30: 5 total, 4 up, 5 in 2024-04-14T13:32:24.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:23 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:32:24.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:23 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:32:24.695 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-14T13:32:24.695 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-14T13:32:24.696 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-04-14T13:32:24.696 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-04-14T13:32:24.696 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121387 s, 864 MB/s 2024-04-14T13:32:24.696 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-04-14T13:32:25.035 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:24 smithi012 ceph-mon[27236]: purged_snaps scrub starts 2024-04-14T13:32:25.036 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:24 smithi012 ceph-mon[27236]: purged_snaps scrub ok 2024-04-14T13:32:25.036 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:24 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:32:25.036 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:24 smithi012 ceph-mon[27236]: osd.4 [v2:172.21.15.12:6800/2473086748,v1:172.21.15.12:6801/2473086748] boot 2024-04-14T13:32:25.036 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:24 smithi012 ceph-mon[27236]: osdmap e31: 5 total, 5 up, 5 in 2024-04-14T13:32:25.036 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:24 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:32:25.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:24 smithi008 ceph-mon[24786]: purged_snaps scrub starts 2024-04-14T13:32:25.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:24 smithi008 ceph-mon[24786]: purged_snaps scrub ok 2024-04-14T13:32:25.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:24 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:32:25.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:24 smithi008 ceph-mon[24786]: osd.4 [v2:172.21.15.12:6800/2473086748,v1:172.21.15.12:6801/2473086748] boot 2024-04-14T13:32:25.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:24 smithi008 ceph-mon[24786]: osdmap e31: 5 total, 5 up, 5 in 2024-04-14T13:32:25.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:24 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:32:25.489 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi012:vg_nvme/lv_3 2024-04-14T13:32:25.731 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:25 smithi012 ceph-mon[27236]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:25.731 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:25 smithi012 ceph-mon[27236]: osdmap e32: 5 total, 5 up, 5 in 2024-04-14T13:32:26.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:25 smithi008 ceph-mon[24786]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:32:26.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:25 smithi008 ceph-mon[24786]: osdmap e32: 5 total, 5 up, 5 in 2024-04-14T13:32:27.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:26 smithi012 ceph-mon[27236]: osdmap e33: 5 total, 5 up, 5 in 2024-04-14T13:32:27.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:26 smithi008 ceph-mon[24786]: osdmap e33: 5 total, 5 up, 5 in 2024-04-14T13:32:27.728 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:27 smithi012 ceph-mon[27236]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:32:27.729 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:27 smithi012 ceph-mon[27236]: osdmap e34: 5 total, 5 up, 5 in 2024-04-14T13:32:27.729 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:27 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:27.729 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:27 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:32:27.729 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:27 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:27.729 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:27 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:27.729 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:27 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:32:27.729 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:27 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:27.729 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:27 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:32:27.729 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:27 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:27.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:27 smithi008 ceph-mon[24786]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:32:27.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:27 smithi008 ceph-mon[24786]: osdmap e34: 5 total, 5 up, 5 in 2024-04-14T13:32:27.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:27 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:27.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:27 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:32:27.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:27 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:27.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:27 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:27.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:27 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:32:27.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:27 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:27.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:27 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:32:27.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:27 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:28.892 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:28 smithi012 ceph-mon[27236]: Detected new or changed devices on smithi012 2024-04-14T13:32:28.893 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:28 smithi012 ceph-mon[27236]: Adjusting osd_memory_target on smithi012 to 15941M 2024-04-14T13:32:28.893 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:28 smithi012 ceph-mon[27236]: osdmap e35: 5 total, 5 up, 5 in 2024-04-14T13:32:28.893 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:28 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:32:28.893 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:28 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:32:28.893 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:28 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:29.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:28 smithi008 ceph-mon[24786]: Detected new or changed devices on smithi012 2024-04-14T13:32:29.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:28 smithi008 ceph-mon[24786]: Adjusting osd_memory_target on smithi012 to 15941M 2024-04-14T13:32:29.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:28 smithi008 ceph-mon[24786]: osdmap e35: 5 total, 5 up, 5 in 2024-04-14T13:32:29.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:28 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:32:29.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:28 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:32:29.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:28 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:29.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:29 smithi012 ceph-mon[27236]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:32:29.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:29 smithi012 ceph-mon[27236]: from='client.24171 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:32:30.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:29 smithi008 ceph-mon[24786]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:32:30.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:29 smithi008 ceph-mon[24786]: from='client.24171 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:32:31.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:31 smithi012 ceph-mon[27236]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-14T13:32:31.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:31 smithi012 ceph-mon[27236]: from='client.? 172.21.15.12:0/2834203366' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a59bc6f0-504a-4c62-8164-2bf98525712a"}]: dispatch 2024-04-14T13:32:31.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:31 smithi012 ceph-mon[27236]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a59bc6f0-504a-4c62-8164-2bf98525712a"}]: dispatch 2024-04-14T13:32:31.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:31 smithi012 ceph-mon[27236]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a59bc6f0-504a-4c62-8164-2bf98525712a"}]': finished 2024-04-14T13:32:31.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:31 smithi012 ceph-mon[27236]: osdmap e36: 6 total, 5 up, 6 in 2024-04-14T13:32:31.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:31 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:32:31.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:31 smithi012 ceph-mon[27236]: from='client.? 172.21.15.12:0/433056454' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:32:31.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:31 smithi008 ceph-mon[24786]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-14T13:32:31.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:31 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/2834203366' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a59bc6f0-504a-4c62-8164-2bf98525712a"}]: dispatch 2024-04-14T13:32:31.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:31 smithi008 ceph-mon[24786]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a59bc6f0-504a-4c62-8164-2bf98525712a"}]: dispatch 2024-04-14T13:32:31.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:31 smithi008 ceph-mon[24786]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a59bc6f0-504a-4c62-8164-2bf98525712a"}]': finished 2024-04-14T13:32:31.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:31 smithi008 ceph-mon[24786]: osdmap e36: 6 total, 5 up, 6 in 2024-04-14T13:32:31.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:31 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:32:31.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:31 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/433056454' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:32:32.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:32:32.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:32:32.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:32:32.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:32:33.703 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:33 smithi012 ceph-mon[27236]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-14T13:32:33.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:33 smithi008 ceph-mon[24786]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-14T13:32:35.626 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:35 smithi012 ceph-mon[27236]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-14T13:32:35.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:35 smithi008 ceph-mon[24786]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-14T13:32:37.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:37 smithi012 ceph-mon[27236]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-14T13:32:37.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:37 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T13:32:37.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:37 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:37.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:37 smithi012 ceph-mon[27236]: Deploying daemon osd.5 on smithi012 2024-04-14T13:32:37.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:37 smithi008 ceph-mon[24786]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-14T13:32:37.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:37 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T13:32:37.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:37 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:37.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:37 smithi008 ceph-mon[24786]: Deploying daemon osd.5 on smithi012 2024-04-14T13:32:39.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:39 smithi012 ceph-mon[27236]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-14T13:32:39.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:39 smithi008 ceph-mon[24786]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-14T13:32:41.730 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:41 smithi012 ceph-mon[27236]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:32:41.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:41 smithi008 ceph-mon[24786]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:32:43.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:42 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:43.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:42 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:32:43.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:42 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:43.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:42 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:32:43.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:42 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:43.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:42 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:32:43.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:42 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:43.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:42 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:32:43.952 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:43 smithi012 ceph-mon[27236]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:32:44.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:43 smithi008 ceph-mon[24786]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:32:45.148 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 5 on host 'smithi012' 2024-04-14T13:32:46.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:45 smithi012 ceph-mon[27236]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:32:46.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:45 smithi012 ceph-mon[27236]: from='osd.5 [v2:172.21.15.12:6808/3659656891,v1:172.21.15.12:6809/3659656891]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T13:32:46.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:45 smithi012 ceph-mon[27236]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T13:32:46.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:45 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:46.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:45 smithi008 ceph-mon[24786]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:32:46.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:45 smithi008 ceph-mon[24786]: from='osd.5 [v2:172.21.15.12:6808/3659656891,v1:172.21.15.12:6809/3659656891]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T13:32:46.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:45 smithi008 ceph-mon[24786]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T13:32:46.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:45 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:46.122 DEBUG:teuthology.orchestra.run.smithi012:osd.5> sudo journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.5.service 2024-04-14T13:32:46.124 INFO:tasks.cephadm:Deploying osd.6 on smithi012 with /dev/vg_nvme/lv_2... 2024-04-14T13:32:46.124 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-14T13:32:46.729 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:46 smithi012 ceph-mon[27236]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-14T13:32:46.729 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:46 smithi012 ceph-mon[27236]: osdmap e37: 6 total, 5 up, 6 in 2024-04-14T13:32:46.729 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:46 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:32:46.730 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:46 smithi012 ceph-mon[27236]: from='osd.5 [v2:172.21.15.12:6808/3659656891,v1:172.21.15.12:6809/3659656891]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:32:46.730 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:46 smithi012 ceph-mon[27236]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:32:46.730 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:46 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:46.730 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:32:46 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[34450]: 2024-04-14T13:32:46.718+0000 7f8d3b99e700 -1 osd.5 0 waiting for initial osdmap 2024-04-14T13:32:46.730 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:32:46 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[34450]: 2024-04-14T13:32:46.726+0000 7f8d37337700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:32:47.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:46 smithi008 ceph-mon[24786]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-14T13:32:47.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:46 smithi008 ceph-mon[24786]: osdmap e37: 6 total, 5 up, 6 in 2024-04-14T13:32:47.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:46 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:32:47.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:46 smithi008 ceph-mon[24786]: from='osd.5 [v2:172.21.15.12:6808/3659656891,v1:172.21.15.12:6809/3659656891]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:32:47.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:46 smithi008 ceph-mon[24786]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:32:47.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:46 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:47.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:47 smithi008 ceph-mon[24786]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:32:47.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:47 smithi008 ceph-mon[24786]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-14T13:32:47.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:47 smithi008 ceph-mon[24786]: osdmap e38: 6 total, 5 up, 6 in 2024-04-14T13:32:47.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:47 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:32:47.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:47 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:32:48.018 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:47 smithi012 ceph-mon[27236]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:32:48.018 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:47 smithi012 ceph-mon[27236]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-14T13:32:48.018 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:47 smithi012 ceph-mon[27236]: osdmap e38: 6 total, 5 up, 6 in 2024-04-14T13:32:48.018 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:47 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:32:48.018 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:47 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:32:48.324 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-14T13:32:48.324 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-14T13:32:48.324 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-04-14T13:32:48.324 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-04-14T13:32:48.325 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144675 s, 725 MB/s 2024-04-14T13:32:48.325 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-04-14T13:32:49.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:48 smithi012 ceph-mon[27236]: purged_snaps scrub starts 2024-04-14T13:32:49.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:48 smithi012 ceph-mon[27236]: purged_snaps scrub ok 2024-04-14T13:32:49.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:48 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:32:49.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:48 smithi012 ceph-mon[27236]: osd.5 [v2:172.21.15.12:6808/3659656891,v1:172.21.15.12:6809/3659656891] boot 2024-04-14T13:32:49.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:48 smithi012 ceph-mon[27236]: osdmap e39: 6 total, 6 up, 6 in 2024-04-14T13:32:49.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:48 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:32:49.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:48 smithi008 ceph-mon[24786]: purged_snaps scrub starts 2024-04-14T13:32:49.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:48 smithi008 ceph-mon[24786]: purged_snaps scrub ok 2024-04-14T13:32:49.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:48 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:32:49.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:48 smithi008 ceph-mon[24786]: osd.5 [v2:172.21.15.12:6808/3659656891,v1:172.21.15.12:6809/3659656891] boot 2024-04-14T13:32:49.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:48 smithi008 ceph-mon[24786]: osdmap e39: 6 total, 6 up, 6 in 2024-04-14T13:32:49.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:48 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:32:49.116 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi012:vg_nvme/lv_2 2024-04-14T13:32:49.738 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:49 smithi012 ceph-mon[27236]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:32:49.738 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:49 smithi012 ceph-mon[27236]: osdmap e40: 6 total, 6 up, 6 in 2024-04-14T13:32:50.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:49 smithi008 ceph-mon[24786]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:32:50.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:49 smithi008 ceph-mon[24786]: osdmap e40: 6 total, 6 up, 6 in 2024-04-14T13:32:51.013 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:50 smithi012 ceph-mon[27236]: osdmap e41: 6 total, 6 up, 6 in 2024-04-14T13:32:51.013 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:50 smithi012 ceph-mon[27236]: osdmap e42: 6 total, 6 up, 6 in 2024-04-14T13:32:51.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:50 smithi008 ceph-mon[24786]: osdmap e41: 6 total, 6 up, 6 in 2024-04-14T13:32:51.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:50 smithi008 ceph-mon[24786]: osdmap e42: 6 total, 6 up, 6 in 2024-04-14T13:32:51.944 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:51 smithi012 ceph-mon[27236]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:32:51.944 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:51 smithi012 ceph-mon[27236]: osdmap e43: 6 total, 6 up, 6 in 2024-04-14T13:32:52.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:51 smithi008 ceph-mon[24786]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:32:52.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:51 smithi008 ceph-mon[24786]: osdmap e43: 6 total, 6 up, 6 in 2024-04-14T13:32:52.959 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:52 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:32:52.959 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:52 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:32:52.959 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:52 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:52.960 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:52 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:52.960 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:52 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:32:52.960 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:52 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:32:52.960 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:52 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:52.960 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:52 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:52.960 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:52 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:32:52.960 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:52 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:52.960 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:52 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:32:52.960 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:52 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:53.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:52 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:32:53.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:52 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:32:53.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:52 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:53.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:52 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:53.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:52 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:32:53.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:52 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:32:53.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:52 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:53.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:52 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:53.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:52 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:32:53.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:52 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:32:53.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:52 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:32:53.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:52 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:32:53.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:53 smithi012 ceph-mon[27236]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:32:53.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:53 smithi012 ceph-mon[27236]: from='client.24189 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:32:53.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:53 smithi012 ceph-mon[27236]: Detected new or changed devices on smithi012 2024-04-14T13:32:53.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:53 smithi012 ceph-mon[27236]: Adjusting osd_memory_target on smithi012 to 7970M 2024-04-14T13:32:54.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:53 smithi008 ceph-mon[24786]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:32:54.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:53 smithi008 ceph-mon[24786]: from='client.24189 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:32:54.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:53 smithi008 ceph-mon[24786]: Detected new or changed devices on smithi012 2024-04-14T13:32:54.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:53 smithi008 ceph-mon[24786]: Adjusting osd_memory_target on smithi012 to 7970M 2024-04-14T13:32:55.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:54 smithi012 ceph-mon[27236]: from='client.? 172.21.15.12:0/1329055162' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6267d07e-3bea-4323-bc72-0e6bc68f3469"}]: dispatch 2024-04-14T13:32:55.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:54 smithi012 ceph-mon[27236]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6267d07e-3bea-4323-bc72-0e6bc68f3469"}]: dispatch 2024-04-14T13:32:55.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:54 smithi012 ceph-mon[27236]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6267d07e-3bea-4323-bc72-0e6bc68f3469"}]': finished 2024-04-14T13:32:55.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:54 smithi012 ceph-mon[27236]: osdmap e44: 7 total, 6 up, 7 in 2024-04-14T13:32:55.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:54 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:32:55.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:54 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/1329055162' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6267d07e-3bea-4323-bc72-0e6bc68f3469"}]: dispatch 2024-04-14T13:32:55.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:54 smithi008 ceph-mon[24786]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6267d07e-3bea-4323-bc72-0e6bc68f3469"}]: dispatch 2024-04-14T13:32:55.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:54 smithi008 ceph-mon[24786]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6267d07e-3bea-4323-bc72-0e6bc68f3469"}]': finished 2024-04-14T13:32:55.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:54 smithi008 ceph-mon[24786]: osdmap e44: 7 total, 6 up, 7 in 2024-04-14T13:32:55.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:54 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:32:56.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:55 smithi012 ceph-mon[27236]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:32:56.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:55 smithi012 ceph-mon[27236]: from='client.? 172.21.15.12:0/2051878434' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:32:56.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:55 smithi008 ceph-mon[24786]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:32:56.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:55 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/2051878434' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:32:57.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:57 smithi008 ceph-mon[24786]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:32:58.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:57 smithi012 ceph-mon[27236]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:33:00.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:32:59 smithi012 ceph-mon[27236]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:33:00.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:32:59 smithi008 ceph-mon[24786]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:33:01.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:01 smithi012 ceph-mon[27236]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:33:01.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:01 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T13:33:01.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:01 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:33:01.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:01 smithi012 ceph-mon[27236]: Deploying daemon osd.6 on smithi012 2024-04-14T13:33:01.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:01 smithi008 ceph-mon[24786]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:33:01.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:01 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T13:33:01.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:01 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:33:01.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:01 smithi008 ceph-mon[24786]: Deploying daemon osd.6 on smithi012 2024-04-14T13:33:03.787 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:03 smithi012 ceph-mon[27236]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:33:03.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:03 smithi008 ceph-mon[24786]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:33:05.665 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:05 smithi012 ceph-mon[27236]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:33:05.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:05 smithi008 ceph-mon[24786]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:33:07.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:07.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:33:07.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:33:07.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:33:07.141 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:07.141 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:33:07.141 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:33:07.141 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:33:08.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:07 smithi012 ceph-mon[27236]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:33:08.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:07 smithi008 ceph-mon[24786]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:33:09.141 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 6 on host 'smithi012' 2024-04-14T13:33:09.786 DEBUG:teuthology.orchestra.run.smithi012:osd.6> sudo journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.6.service 2024-04-14T13:33:09.788 INFO:tasks.cephadm:Deploying osd.7 on smithi012 with /dev/vg_nvme/lv_1... 2024-04-14T13:33:09.788 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-14T13:33:10.043 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:09 smithi012 ceph-mon[27236]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:33:10.043 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:09 smithi012 ceph-mon[27236]: from='osd.6 [v2:172.21.15.12:6816/2575290587,v1:172.21.15.12:6817/2575290587]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T13:33:10.043 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:09 smithi012 ceph-mon[27236]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T13:33:10.043 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:09 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:10.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:09 smithi008 ceph-mon[24786]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:33:10.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:09 smithi008 ceph-mon[24786]: from='osd.6 [v2:172.21.15.12:6816/2575290587,v1:172.21.15.12:6817/2575290587]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T13:33:10.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:09 smithi008 ceph-mon[24786]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T13:33:10.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:09 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:10.920 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:10 smithi012 ceph-mon[27236]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-14T13:33:10.920 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:10 smithi012 ceph-mon[27236]: osdmap e45: 7 total, 6 up, 7 in 2024-04-14T13:33:10.920 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:10 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:33:10.920 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:10 smithi012 ceph-mon[27236]: from='osd.6 [v2:172.21.15.12:6816/2575290587,v1:172.21.15.12:6817/2575290587]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:33:10.920 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:10 smithi012 ceph-mon[27236]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:33:10.920 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:33:10 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[37415]: 2024-04-14T13:33:10.833+0000 7f3313827700 -1 osd.6 0 waiting for initial osdmap 2024-04-14T13:33:10.921 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:33:10 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[37415]: 2024-04-14T13:33:10.840+0000 7f330d1bc700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:33:11.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:10 smithi008 ceph-mon[24786]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-14T13:33:11.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:10 smithi008 ceph-mon[24786]: osdmap e45: 7 total, 6 up, 7 in 2024-04-14T13:33:11.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:10 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:33:11.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:10 smithi008 ceph-mon[24786]: from='osd.6 [v2:172.21.15.12:6816/2575290587,v1:172.21.15.12:6817/2575290587]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:33:11.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:10 smithi008 ceph-mon[24786]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:33:12.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:11 smithi008 ceph-mon[24786]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:33:12.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:11 smithi008 ceph-mon[24786]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-14T13:33:12.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:11 smithi008 ceph-mon[24786]: osdmap e46: 7 total, 6 up, 7 in 2024-04-14T13:33:12.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:11 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:33:12.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:11 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:33:12.267 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:11 smithi012 ceph-mon[27236]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:33:12.267 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:11 smithi012 ceph-mon[27236]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-14T13:33:12.267 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:11 smithi012 ceph-mon[27236]: osdmap e46: 7 total, 6 up, 7 in 2024-04-14T13:33:12.267 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:11 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:33:12.267 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:11 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:33:12.374 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-14T13:33:12.375 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-14T13:33:12.375 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-04-14T13:33:12.375 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-04-14T13:33:12.375 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0139757 s, 750 MB/s 2024-04-14T13:33:12.375 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-04-14T13:33:13.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:12 smithi012 ceph-mon[27236]: purged_snaps scrub starts 2024-04-14T13:33:13.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:12 smithi012 ceph-mon[27236]: purged_snaps scrub ok 2024-04-14T13:33:13.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:12 smithi012 ceph-mon[27236]: osd.6 [v2:172.21.15.12:6816/2575290587,v1:172.21.15.12:6817/2575290587] boot 2024-04-14T13:33:13.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:12 smithi012 ceph-mon[27236]: osdmap e47: 7 total, 7 up, 7 in 2024-04-14T13:33:13.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:12 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:33:13.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:12 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:13.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:12 smithi012 ceph-mon[27236]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:33:13.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:12 smithi012 ceph-mon[27236]: osdmap e48: 7 total, 7 up, 7 in 2024-04-14T13:33:13.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:12 smithi008 ceph-mon[24786]: purged_snaps scrub starts 2024-04-14T13:33:13.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:12 smithi008 ceph-mon[24786]: purged_snaps scrub ok 2024-04-14T13:33:13.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:12 smithi008 ceph-mon[24786]: osd.6 [v2:172.21.15.12:6816/2575290587,v1:172.21.15.12:6817/2575290587] boot 2024-04-14T13:33:13.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:12 smithi008 ceph-mon[24786]: osdmap e47: 7 total, 7 up, 7 in 2024-04-14T13:33:13.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:12 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:33:13.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:12 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:13.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:12 smithi008 ceph-mon[24786]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:33:13.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:12 smithi008 ceph-mon[24786]: osdmap e48: 7 total, 7 up, 7 in 2024-04-14T13:33:13.488 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi012:vg_nvme/lv_1 2024-04-14T13:33:15.095 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:14 smithi012 ceph-mon[27236]: osdmap e49: 7 total, 7 up, 7 in 2024-04-14T13:33:15.095 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:14 smithi012 ceph-mon[27236]: pgmap v148: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:33:15.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:14 smithi008 ceph-mon[24786]: osdmap e49: 7 total, 7 up, 7 in 2024-04-14T13:33:15.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:14 smithi008 ceph-mon[24786]: pgmap v148: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:33:16.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:15 smithi012 ceph-mon[27236]: osdmap e50: 7 total, 7 up, 7 in 2024-04-14T13:33:16.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:15 smithi008 ceph-mon[24786]: osdmap e50: 7 total, 7 up, 7 in 2024-04-14T13:33:17.043 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:16 smithi012 ceph-mon[27236]: osdmap e51: 7 total, 7 up, 7 in 2024-04-14T13:33:17.043 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:16 smithi012 ceph-mon[27236]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:33:17.043 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:16 smithi012 ceph-mon[27236]: Detected new or changed devices on smithi012 2024-04-14T13:33:17.043 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:16 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:17.043 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:16 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:33:17.043 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:16 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:33:17.043 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:16 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:33:17.043 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:16 smithi012 ceph-mon[27236]: Adjusting osd_memory_target on smithi012 to 5313M 2024-04-14T13:33:17.043 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:16 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:17.043 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:16 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:17.043 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:16 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:33:17.044 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:16 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:33:17.044 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:16 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:33:17.044 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:16 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:17.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:16 smithi008 ceph-mon[24786]: osdmap e51: 7 total, 7 up, 7 in 2024-04-14T13:33:17.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:16 smithi008 ceph-mon[24786]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:33:17.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:16 smithi008 ceph-mon[24786]: Detected new or changed devices on smithi012 2024-04-14T13:33:17.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:16 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:17.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:16 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:33:17.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:16 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:33:17.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:16 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:33:17.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:16 smithi008 ceph-mon[24786]: Adjusting osd_memory_target on smithi012 to 5313M 2024-04-14T13:33:17.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:16 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:17.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:16 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:17.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:16 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:33:17.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:16 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:33:17.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:16 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:33:17.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:16 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:18.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:17 smithi012 ceph-mon[27236]: from='client.24207 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:33:18.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:17 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:33:18.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:17 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:33:18.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:17 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:33:18.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:17 smithi008 ceph-mon[24786]: from='client.24207 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:33:18.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:17 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:33:18.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:17 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:33:18.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:17 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:33:19.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:18 smithi012 ceph-mon[27236]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-14T13:33:19.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:18 smithi008 ceph-mon[24786]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-14T13:33:20.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:19 smithi012 ceph-mon[27236]: from='client.? 172.21.15.12:0/3479519602' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c59bc81-f988-4f95-96f0-be7924acb6df"}]: dispatch 2024-04-14T13:33:20.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:19 smithi012 ceph-mon[27236]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c59bc81-f988-4f95-96f0-be7924acb6df"}]: dispatch 2024-04-14T13:33:20.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:19 smithi012 ceph-mon[27236]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c59bc81-f988-4f95-96f0-be7924acb6df"}]': finished 2024-04-14T13:33:20.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:19 smithi012 ceph-mon[27236]: osdmap e52: 8 total, 7 up, 8 in 2024-04-14T13:33:20.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:19 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:33:20.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:19 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/3479519602' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c59bc81-f988-4f95-96f0-be7924acb6df"}]: dispatch 2024-04-14T13:33:20.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:19 smithi008 ceph-mon[24786]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c59bc81-f988-4f95-96f0-be7924acb6df"}]: dispatch 2024-04-14T13:33:20.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:19 smithi008 ceph-mon[24786]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c59bc81-f988-4f95-96f0-be7924acb6df"}]': finished 2024-04-14T13:33:20.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:19 smithi008 ceph-mon[24786]: osdmap e52: 8 total, 7 up, 8 in 2024-04-14T13:33:20.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:19 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:33:21.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:20 smithi012 ceph-mon[27236]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-14T13:33:21.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:20 smithi012 ceph-mon[27236]: from='client.? 172.21.15.12:0/3061918201' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:33:21.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:20 smithi008 ceph-mon[24786]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-14T13:33:21.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:20 smithi008 ceph-mon[24786]: from='client.? 172.21.15.12:0/3061918201' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:33:23.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:22 smithi012 ceph-mon[27236]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-14T13:33:23.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:22 smithi008 ceph-mon[24786]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-14T13:33:25.206 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:24 smithi012 ceph-mon[27236]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-14T13:33:25.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:24 smithi008 ceph-mon[24786]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-14T13:33:26.640 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:26 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T13:33:26.640 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:26 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:33:26.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:26 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T13:33:26.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:26 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:33:27.617 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:27 smithi012 ceph-mon[27236]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-14T13:33:27.617 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:27 smithi012 ceph-mon[27236]: Deploying daemon osd.7 on smithi012 2024-04-14T13:33:27.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:27 smithi008 ceph-mon[24786]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-14T13:33:27.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:27 smithi008 ceph-mon[24786]: Deploying daemon osd.7 on smithi012 2024-04-14T13:33:29.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:29 smithi012 ceph-mon[27236]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:33:29.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:29 smithi008 ceph-mon[24786]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:33:31.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:31 smithi012 ceph-mon[27236]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:33:31.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:31 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:31.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:31 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:33:31.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:31 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:33:31.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:31 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:33:31.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:31 smithi008 ceph-mon[24786]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:33:31.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:31 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:31.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:31 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:33:31.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:31 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:33:31.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:31 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:33:32.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:33:32.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:33:32.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:33:32.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:33:33.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:33 smithi012 ceph-mon[27236]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:33:33.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:33 smithi008 ceph-mon[24786]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:33:34.371 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 7 on host 'smithi012' 2024-04-14T13:33:34.676 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:34 smithi012 ceph-mon[27236]: from='osd.7 [v2:172.21.15.12:6824/3762015288,v1:172.21.15.12:6825/3762015288]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T13:33:34.677 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:34 smithi012 ceph-mon[27236]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T13:33:34.677 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:34 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:34.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:34 smithi008 ceph-mon[24786]: from='osd.7 [v2:172.21.15.12:6824/3762015288,v1:172.21.15.12:6825/3762015288]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T13:33:34.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:34 smithi008 ceph-mon[24786]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T13:33:34.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:34 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:35.083 DEBUG:teuthology.orchestra.run.smithi012:osd.7> sudo journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.7.service 2024-04-14T13:33:35.086 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-14T13:33:35.086 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph osd stat -f json 2024-04-14T13:33:35.639 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:35 smithi012 ceph-mon[27236]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:33:35.639 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:35 smithi012 ceph-mon[27236]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-14T13:33:35.639 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:35 smithi012 ceph-mon[27236]: osdmap e53: 8 total, 7 up, 8 in 2024-04-14T13:33:35.639 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:35 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:33:35.639 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:35 smithi012 ceph-mon[27236]: from='osd.7 [v2:172.21.15.12:6824/3762015288,v1:172.21.15.12:6825/3762015288]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:33:35.639 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:35 smithi012 ceph-mon[27236]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:33:35.639 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:33:35 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[40426]: 2024-04-14T13:33:35.549+0000 7fbfdcaa9700 -1 osd.7 0 waiting for initial osdmap 2024-04-14T13:33:35.640 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:33:35 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[40426]: 2024-04-14T13:33:35.556+0000 7fbfd5c3d700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:33:35.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:35 smithi008 ceph-mon[24786]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:33:35.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:35 smithi008 ceph-mon[24786]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-14T13:33:35.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:35 smithi008 ceph-mon[24786]: osdmap e53: 8 total, 7 up, 8 in 2024-04-14T13:33:35.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:35 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:33:35.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:35 smithi008 ceph-mon[24786]: from='osd.7 [v2:172.21.15.12:6824/3762015288,v1:172.21.15.12:6825/3762015288]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:33:35.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:35 smithi008 ceph-mon[24786]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:33:36.714 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:36 smithi012 ceph-mon[27236]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-14T13:33:36.714 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:36 smithi012 ceph-mon[27236]: osdmap e54: 8 total, 7 up, 8 in 2024-04-14T13:33:36.714 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:36 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:33:36.714 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:36 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:33:36.714 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:36 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:36.714 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:36 smithi012 ceph-mon[27236]: osd.7 [v2:172.21.15.12:6824/3762015288,v1:172.21.15.12:6825/3762015288] boot 2024-04-14T13:33:36.715 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:36 smithi012 ceph-mon[27236]: osdmap e55: 8 total, 8 up, 8 in 2024-04-14T13:33:36.715 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:36 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:33:36.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:36 smithi008 ceph-mon[24786]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-14T13:33:36.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:36 smithi008 ceph-mon[24786]: osdmap e54: 8 total, 7 up, 8 in 2024-04-14T13:33:36.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:36 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:33:36.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:36 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:33:36.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:36 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:36.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:36 smithi008 ceph-mon[24786]: osd.7 [v2:172.21.15.12:6824/3762015288,v1:172.21.15.12:6825/3762015288] boot 2024-04-14T13:33:36.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:36 smithi008 ceph-mon[24786]: osdmap e55: 8 total, 8 up, 8 in 2024-04-14T13:33:36.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:36 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:33:37.113 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:33:37.713 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:37 smithi012 ceph-mon[27236]: purged_snaps scrub starts 2024-04-14T13:33:37.713 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:37 smithi012 ceph-mon[27236]: purged_snaps scrub ok 2024-04-14T13:33:37.713 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:37 smithi012 ceph-mon[27236]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:33:37.713 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:37 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/2093161372' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-14T13:33:37.713 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:37 smithi012 ceph-mon[27236]: osdmap e56: 8 total, 8 up, 8 in 2024-04-14T13:33:37.811 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:37 smithi008 ceph-mon[24786]: purged_snaps scrub starts 2024-04-14T13:33:37.811 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:37 smithi008 ceph-mon[24786]: purged_snaps scrub ok 2024-04-14T13:33:37.811 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:37 smithi008 ceph-mon[24786]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:33:37.811 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:37 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2093161372' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-14T13:33:37.811 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:37 smithi008 ceph-mon[24786]: osdmap e56: 8 total, 8 up, 8 in 2024-04-14T13:33:37.832 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1713101616,"num_in_osds":8,"osd_in_since":1713101599,"num_remapped_pgs":1} 2024-04-14T13:33:37.833 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph osd dump --format=json 2024-04-14T13:33:38.054 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:33:39.785 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:33:39.785 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":58,"fsid":"a8b03f6a-fa62-11ee-bc8e-c7b262605968","created":"2024-04-14T13:27:30.263026+0000","modified":"2024-04-14T13:33:39.553949+0000","last_up_change":"2024-04-14T13:33:36.539780+0000","last_in_change":"2024-04-14T13:33:19.341323+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T13:31:34.154843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"4967d1e8-5e91-4e77-b6eb-28d0d9af8475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":1782930059},{"type":"v1","addr":"172.21.15.8:6803","nonce":1782930059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":1782930059},{"type":"v1","addr":"172.21.15.8:6805","nonce":1782930059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":1782930059},{"type":"v1","addr":"172.21.15.8:6809","nonce":1782930059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":1782930059},{"type":"v1","addr":"172.21.15.8:6807","nonce":1782930059}]},"public_addr":"172.21.15.8:6803/1782930059","cluster_addr":"172.21.15.8:6805/1782930059","heartbeat_back_addr":"172.21.15.8:6809/1782930059","heartbeat_front_addr":"172.21.15.8:6807/1782930059","state":["exists","up"]},{"osd":1,"uuid":"64039dfc-a581-4224-82ea-bff2b958dc1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":2761200318},{"type":"v1","addr":"172.21.15.8:6811","nonce":2761200318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":2761200318},{"type":"v1","addr":"172.21.15.8:6813","nonce":2761200318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":2761200318},{"type":"v1","addr":"172.21.15.8:6817","nonce":2761200318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":2761200318},{"type":"v1","addr":"172.21.15.8:6815","nonce":2761200318}]},"public_addr":"172.21.15.8:6811/2761200318","cluster_addr":"172.21.15.8:6813/2761200318","heartbeat_back_addr":"172.21.15.8:6817/2761200318","heartbeat_front_addr":"172.21.15.8:6815/2761200318","state":["exists","up"]},{"osd":2,"uuid":"ee8d332d-b536-4dfe-b0a2-f43d79149677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":3582626304},{"type":"v1","addr":"172.21.15.8:6819","nonce":3582626304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":3582626304},{"type":"v1","addr":"172.21.15.8:6821","nonce":3582626304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":3582626304},{"type":"v1","addr":"172.21.15.8:6825","nonce":3582626304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":3582626304},{"type":"v1","addr":"172.21.15.8:6823","nonce":3582626304}]},"public_addr":"172.21.15.8:6819/3582626304","cluster_addr":"172.21.15.8:6821/3582626304","heartbeat_back_addr":"172.21.15.8:6825/3582626304","heartbeat_front_addr":"172.21.15.8:6823/3582626304","state":["exists","up"]},{"osd":3,"uuid":"65bead61-5127-4144-8704-41a61d19aac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":2354457054},{"type":"v1","addr":"172.21.15.8:6827","nonce":2354457054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":2354457054},{"type":"v1","addr":"172.21.15.8:6829","nonce":2354457054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6832","nonce":2354457054},{"type":"v1","addr":"172.21.15.8:6833","nonce":2354457054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":2354457054},{"type":"v1","addr":"172.21.15.8:6831","nonce":2354457054}]},"public_addr":"172.21.15.8:6827/2354457054","cluster_addr":"172.21.15.8:6829/2354457054","heartbeat_back_addr":"172.21.15.8:6833/2354457054","heartbeat_front_addr":"172.21.15.8:6831/2354457054","state":["exists","up"]},{"osd":4,"uuid":"e81bc030-196f-4623-9d12-2bc124c12c1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2473086748},{"type":"v1","addr":"172.21.15.12:6801","nonce":2473086748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2473086748},{"type":"v1","addr":"172.21.15.12:6803","nonce":2473086748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2473086748},{"type":"v1","addr":"172.21.15.12:6807","nonce":2473086748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2473086748},{"type":"v1","addr":"172.21.15.12:6805","nonce":2473086748}]},"public_addr":"172.21.15.12:6801/2473086748","cluster_addr":"172.21.15.12:6803/2473086748","heartbeat_back_addr":"172.21.15.12:6807/2473086748","heartbeat_front_addr":"172.21.15.12:6805/2473086748","state":["exists","up"]},{"osd":5,"uuid":"a59bc6f0-504a-4c62-8164-2bf98525712a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3659656891},{"type":"v1","addr":"172.21.15.12:6809","nonce":3659656891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3659656891},{"type":"v1","addr":"172.21.15.12:6811","nonce":3659656891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3659656891},{"type":"v1","addr":"172.21.15.12:6815","nonce":3659656891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3659656891},{"type":"v1","addr":"172.21.15.12:6813","nonce":3659656891}]},"public_addr":"172.21.15.12:6809/3659656891","cluster_addr":"172.21.15.12:6811/3659656891","heartbeat_back_addr":"172.21.15.12:6815/3659656891","heartbeat_front_addr":"172.21.15.12:6813/3659656891","state":["exists","up"]},{"osd":6,"uuid":"6267d07e-3bea-4323-bc72-0e6bc68f3469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2575290587},{"type":"v1","addr":"172.21.15.12:6817","nonce":2575290587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":2575290587},{"type":"v1","addr":"172.21.15.12:6819","nonce":2575290587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":2575290587},{"type":"v1","addr":"172.21.15.12:6823","nonce":2575290587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2575290587},{"type":"v1","addr":"172.21.15.12:6821","nonce":2575290587}]},"public_addr":"172.21.15.12:6817/2575290587","cluster_addr":"172.21.15.12:6819/2575290587","heartbeat_back_addr":"172.21.15.12:6823/2575290587","heartbeat_front_addr":"172.21.15.12:6821/2575290587","state":["exists","up"]},{"osd":7,"uuid":"3c59bc81-f988-4f95-96f0-be7924acb6df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3762015288},{"type":"v1","addr":"172.21.15.12:6825","nonce":3762015288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3762015288},{"type":"v1","addr":"172.21.15.12:6827","nonce":3762015288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3762015288},{"type":"v1","addr":"172.21.15.12:6831","nonce":3762015288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3762015288},{"type":"v1","addr":"172.21.15.12:6829","nonce":3762015288}]},"public_addr":"172.21.15.12:6825/3762015288","cluster_addr":"172.21.15.12:6827/3762015288","heartbeat_back_addr":"172.21.15.12:6831/3762015288","heartbeat_front_addr":"172.21.15.12:6829/3762015288","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:30:38.696430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:31:07.057278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:31:32.330617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:32:00.215681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:32:21.755098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:32:45.772849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:33:09.901851+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:33:35.261230+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.8:0/4061795296":"2024-04-15T13:29:31.790648+0000","172.21.15.8:6800/1546778513":"2024-04-15T13:29:31.790648+0000","172.21.15.8:0/3432677301":"2024-04-15T13:28:35.306236+0000","172.21.15.8:6801/1192676720":"2024-04-15T13:28:35.306236+0000","172.21.15.8:6800/1192676720":"2024-04-15T13:28:35.306236+0000","172.21.15.8:0/3050160566":"2024-04-15T13:29:31.790648+0000","172.21.15.8:0/1865709161":"2024-04-15T13:28:35.306236+0000","172.21.15.8:0/4035696983":"2024-04-15T13:28:35.306236+0000","172.21.15.8:6800/596187414":"2024-04-15T13:28:05.823103+0000","172.21.15.8:6801/596187414":"2024-04-15T13:28:05.823103+0000","172.21.15.8:0/1824349976":"2024-04-15T13:28:05.823103+0000","172.21.15.8:0/1159808486":"2024-04-15T13:29:31.790648+0000","172.21.15.8:0/2433183361":"2024-04-15T13:28:05.823103+0000","172.21.15.8:6801/1546778513":"2024-04-15T13:29:31.790648+0000","172.21.15.8:0/1505809637":"2024-04-15T13:28:05.823103+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T13:33:39.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:39 smithi008 ceph-mon[24786]: pgmap v167: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:33:39.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:39 smithi008 ceph-mon[24786]: osdmap e57: 8 total, 8 up, 8 in 2024-04-14T13:33:39.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:39.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:33:39.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:33:39.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:33:39.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:33:39.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:39.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:39.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:33:39.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:33:39.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:33:39.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:39.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:39 smithi012 ceph-mon[27236]: pgmap v167: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:33:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:39 smithi012 ceph-mon[27236]: osdmap e57: 8 total, 8 up, 8 in 2024-04-14T13:33:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:39 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:39 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:33:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:39 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:33:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:39 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:33:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:39 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:33:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:39 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:39 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:39 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:33:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:39 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:33:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:39 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:33:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:39 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:33:40.496 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-14T13:31:34.154843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-14T13:33:40.497 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-14T13:33:40.723 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:33:40.844 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:40 smithi008 ceph-mon[24786]: Detected new or changed devices on smithi012 2024-04-14T13:33:40.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:40 smithi008 ceph-mon[24786]: Adjusting osd_memory_target on smithi012 to 3985M 2024-04-14T13:33:40.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:40 smithi008 ceph-mon[24786]: osdmap e58: 8 total, 8 up, 8 in 2024-04-14T13:33:40.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:40 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/552601055' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T13:33:41.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:40 smithi012 ceph-mon[27236]: Detected new or changed devices on smithi012 2024-04-14T13:33:41.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:40 smithi012 ceph-mon[27236]: Adjusting osd_memory_target on smithi012 to 3985M 2024-04-14T13:33:41.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:40 smithi012 ceph-mon[27236]: osdmap e58: 8 total, 8 up, 8 in 2024-04-14T13:33:41.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:40 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/552601055' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T13:33:41.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:41 smithi008 ceph-mon[24786]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:33:41.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:41 smithi008 ceph-mon[24786]: osdmap e59: 8 total, 8 up, 8 in 2024-04-14T13:33:42.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:41 smithi012 ceph-mon[27236]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:33:42.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:41 smithi012 ceph-mon[27236]: osdmap e59: 8 total, 8 up, 8 in 2024-04-14T13:33:42.585 INFO:teuthology.orchestra.run.smithi008.stdout:pg_num: 1 2024-04-14T13:33:43.270 INFO:tasks.cephadm:Setting up client nodes... 2024-04-14T13:33:43.271 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-14T13:33:43.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:43 smithi008 ceph-mon[24786]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-04-14T13:33:43.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:43 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/4153262392' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-14T13:33:44.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:43 smithi012 ceph-mon[27236]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-04-14T13:33:44.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:43 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/4153262392' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-14T13:33:45.358 INFO:teuthology.orchestra.run.smithi008.stdout:[client.0] 2024-04-14T13:33:45.358 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQA52xtmBu7wFBAAxX2b74jNlUaljhEFI6z9Ag== 2024-04-14T13:33:45.692 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:45 smithi008 ceph-mon[24786]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-14T13:33:45.692 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:45 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/3546544248' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T13:33:45.692 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:45 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/3546544248' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T13:33:45.852 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:45 smithi012 ceph-mon[27236]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-14T13:33:45.852 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:45 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/3546544248' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T13:33:45.852 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:45 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/3546544248' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T13:33:46.099 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-14T13:33:46.099 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-14T13:33:46.099 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-14T13:33:46.143 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-14T13:33:46.144 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-14T13:33:46.144 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph mgr dump --format=json 2024-04-14T13:33:46.401 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:33:47.800 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:47 smithi008 ceph-mon[24786]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-14T13:33:48.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:47 smithi012 ceph-mon[27236]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-14T13:33:48.260 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:33:48.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:48 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/1640281204' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-14T13:33:48.961 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":19,"active_gid":14206,"active_name":"smithi008.yemlhr","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":2171605416},{"type":"v1","addr":"172.21.15.8:6801","nonce":2171605416}]},"active_addr":"172.21.15.8:6801/2171605416","active_change":"2024-04-14T13:29:31.790933+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":14226,"name":"smithi012.ksvnxm","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.8:8443/","prometheus":"http://172.21.15.8:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":3253665015}]},{"addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":1553580489}]},{"addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":253971711}]},{"addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":739686773}]}]}} 2024-04-14T13:33:48.967 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-14T13:33:48.967 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-14T13:33:48.967 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph osd dump --format=json 2024-04-14T13:33:48.979 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:48 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/1640281204' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-14T13:33:49.186 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:33:49.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:49 smithi008 ceph-mon[24786]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-14T13:33:50.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:49 smithi012 ceph-mon[27236]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-14T13:33:51.067 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:33:51.067 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":59,"fsid":"a8b03f6a-fa62-11ee-bc8e-c7b262605968","created":"2024-04-14T13:27:30.263026+0000","modified":"2024-04-14T13:33:40.555145+0000","last_up_change":"2024-04-14T13:33:36.539780+0000","last_in_change":"2024-04-14T13:33:19.341323+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T13:31:34.154843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"4967d1e8-5e91-4e77-b6eb-28d0d9af8475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":1782930059},{"type":"v1","addr":"172.21.15.8:6803","nonce":1782930059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":1782930059},{"type":"v1","addr":"172.21.15.8:6805","nonce":1782930059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":1782930059},{"type":"v1","addr":"172.21.15.8:6809","nonce":1782930059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":1782930059},{"type":"v1","addr":"172.21.15.8:6807","nonce":1782930059}]},"public_addr":"172.21.15.8:6803/1782930059","cluster_addr":"172.21.15.8:6805/1782930059","heartbeat_back_addr":"172.21.15.8:6809/1782930059","heartbeat_front_addr":"172.21.15.8:6807/1782930059","state":["exists","up"]},{"osd":1,"uuid":"64039dfc-a581-4224-82ea-bff2b958dc1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":2761200318},{"type":"v1","addr":"172.21.15.8:6811","nonce":2761200318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":2761200318},{"type":"v1","addr":"172.21.15.8:6813","nonce":2761200318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":2761200318},{"type":"v1","addr":"172.21.15.8:6817","nonce":2761200318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":2761200318},{"type":"v1","addr":"172.21.15.8:6815","nonce":2761200318}]},"public_addr":"172.21.15.8:6811/2761200318","cluster_addr":"172.21.15.8:6813/2761200318","heartbeat_back_addr":"172.21.15.8:6817/2761200318","heartbeat_front_addr":"172.21.15.8:6815/2761200318","state":["exists","up"]},{"osd":2,"uuid":"ee8d332d-b536-4dfe-b0a2-f43d79149677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":3582626304},{"type":"v1","addr":"172.21.15.8:6819","nonce":3582626304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":3582626304},{"type":"v1","addr":"172.21.15.8:6821","nonce":3582626304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":3582626304},{"type":"v1","addr":"172.21.15.8:6825","nonce":3582626304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":3582626304},{"type":"v1","addr":"172.21.15.8:6823","nonce":3582626304}]},"public_addr":"172.21.15.8:6819/3582626304","cluster_addr":"172.21.15.8:6821/3582626304","heartbeat_back_addr":"172.21.15.8:6825/3582626304","heartbeat_front_addr":"172.21.15.8:6823/3582626304","state":["exists","up"]},{"osd":3,"uuid":"65bead61-5127-4144-8704-41a61d19aac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":2354457054},{"type":"v1","addr":"172.21.15.8:6827","nonce":2354457054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":2354457054},{"type":"v1","addr":"172.21.15.8:6829","nonce":2354457054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6832","nonce":2354457054},{"type":"v1","addr":"172.21.15.8:6833","nonce":2354457054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":2354457054},{"type":"v1","addr":"172.21.15.8:6831","nonce":2354457054}]},"public_addr":"172.21.15.8:6827/2354457054","cluster_addr":"172.21.15.8:6829/2354457054","heartbeat_back_addr":"172.21.15.8:6833/2354457054","heartbeat_front_addr":"172.21.15.8:6831/2354457054","state":["exists","up"]},{"osd":4,"uuid":"e81bc030-196f-4623-9d12-2bc124c12c1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2473086748},{"type":"v1","addr":"172.21.15.12:6801","nonce":2473086748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2473086748},{"type":"v1","addr":"172.21.15.12:6803","nonce":2473086748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2473086748},{"type":"v1","addr":"172.21.15.12:6807","nonce":2473086748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2473086748},{"type":"v1","addr":"172.21.15.12:6805","nonce":2473086748}]},"public_addr":"172.21.15.12:6801/2473086748","cluster_addr":"172.21.15.12:6803/2473086748","heartbeat_back_addr":"172.21.15.12:6807/2473086748","heartbeat_front_addr":"172.21.15.12:6805/2473086748","state":["exists","up"]},{"osd":5,"uuid":"a59bc6f0-504a-4c62-8164-2bf98525712a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3659656891},{"type":"v1","addr":"172.21.15.12:6809","nonce":3659656891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3659656891},{"type":"v1","addr":"172.21.15.12:6811","nonce":3659656891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3659656891},{"type":"v1","addr":"172.21.15.12:6815","nonce":3659656891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3659656891},{"type":"v1","addr":"172.21.15.12:6813","nonce":3659656891}]},"public_addr":"172.21.15.12:6809/3659656891","cluster_addr":"172.21.15.12:6811/3659656891","heartbeat_back_addr":"172.21.15.12:6815/3659656891","heartbeat_front_addr":"172.21.15.12:6813/3659656891","state":["exists","up"]},{"osd":6,"uuid":"6267d07e-3bea-4323-bc72-0e6bc68f3469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2575290587},{"type":"v1","addr":"172.21.15.12:6817","nonce":2575290587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":2575290587},{"type":"v1","addr":"172.21.15.12:6819","nonce":2575290587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":2575290587},{"type":"v1","addr":"172.21.15.12:6823","nonce":2575290587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2575290587},{"type":"v1","addr":"172.21.15.12:6821","nonce":2575290587}]},"public_addr":"172.21.15.12:6817/2575290587","cluster_addr":"172.21.15.12:6819/2575290587","heartbeat_back_addr":"172.21.15.12:6823/2575290587","heartbeat_front_addr":"172.21.15.12:6821/2575290587","state":["exists","up"]},{"osd":7,"uuid":"3c59bc81-f988-4f95-96f0-be7924acb6df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3762015288},{"type":"v1","addr":"172.21.15.12:6825","nonce":3762015288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3762015288},{"type":"v1","addr":"172.21.15.12:6827","nonce":3762015288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3762015288},{"type":"v1","addr":"172.21.15.12:6831","nonce":3762015288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3762015288},{"type":"v1","addr":"172.21.15.12:6829","nonce":3762015288}]},"public_addr":"172.21.15.12:6825/3762015288","cluster_addr":"172.21.15.12:6827/3762015288","heartbeat_back_addr":"172.21.15.12:6831/3762015288","heartbeat_front_addr":"172.21.15.12:6829/3762015288","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:30:38.696430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:31:07.057278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:31:32.330617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:32:00.215681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:32:21.755098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:32:45.772849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:33:09.901851+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:33:35.261230+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.8:0/4061795296":"2024-04-15T13:29:31.790648+0000","172.21.15.8:6800/1546778513":"2024-04-15T13:29:31.790648+0000","172.21.15.8:0/3432677301":"2024-04-15T13:28:35.306236+0000","172.21.15.8:6801/1192676720":"2024-04-15T13:28:35.306236+0000","172.21.15.8:6800/1192676720":"2024-04-15T13:28:35.306236+0000","172.21.15.8:0/3050160566":"2024-04-15T13:29:31.790648+0000","172.21.15.8:0/1865709161":"2024-04-15T13:28:35.306236+0000","172.21.15.8:0/4035696983":"2024-04-15T13:28:35.306236+0000","172.21.15.8:6800/596187414":"2024-04-15T13:28:05.823103+0000","172.21.15.8:6801/596187414":"2024-04-15T13:28:05.823103+0000","172.21.15.8:0/1824349976":"2024-04-15T13:28:05.823103+0000","172.21.15.8:0/1159808486":"2024-04-15T13:29:31.790648+0000","172.21.15.8:0/2433183361":"2024-04-15T13:28:05.823103+0000","172.21.15.8:6801/1546778513":"2024-04-15T13:29:31.790648+0000","172.21.15.8:0/1505809637":"2024-04-15T13:28:05.823103+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T13:33:51.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:51 smithi008 ceph-mon[24786]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-14T13:33:51.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:51 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/787746192' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T13:33:51.744 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-14T13:33:51.744 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph osd dump --format=json 2024-04-14T13:33:51.967 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:33:52.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:51 smithi012 ceph-mon[27236]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-14T13:33:52.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:51 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/787746192' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T13:33:53.800 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:33:53.801 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:53 smithi008 ceph-mon[24786]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s, 0 objects/s recovering 2024-04-14T13:33:53.801 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":59,"fsid":"a8b03f6a-fa62-11ee-bc8e-c7b262605968","created":"2024-04-14T13:27:30.263026+0000","modified":"2024-04-14T13:33:40.555145+0000","last_up_change":"2024-04-14T13:33:36.539780+0000","last_in_change":"2024-04-14T13:33:19.341323+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T13:31:34.154843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"4967d1e8-5e91-4e77-b6eb-28d0d9af8475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":1782930059},{"type":"v1","addr":"172.21.15.8:6803","nonce":1782930059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":1782930059},{"type":"v1","addr":"172.21.15.8:6805","nonce":1782930059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":1782930059},{"type":"v1","addr":"172.21.15.8:6809","nonce":1782930059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":1782930059},{"type":"v1","addr":"172.21.15.8:6807","nonce":1782930059}]},"public_addr":"172.21.15.8:6803/1782930059","cluster_addr":"172.21.15.8:6805/1782930059","heartbeat_back_addr":"172.21.15.8:6809/1782930059","heartbeat_front_addr":"172.21.15.8:6807/1782930059","state":["exists","up"]},{"osd":1,"uuid":"64039dfc-a581-4224-82ea-bff2b958dc1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":2761200318},{"type":"v1","addr":"172.21.15.8:6811","nonce":2761200318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":2761200318},{"type":"v1","addr":"172.21.15.8:6813","nonce":2761200318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":2761200318},{"type":"v1","addr":"172.21.15.8:6817","nonce":2761200318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":2761200318},{"type":"v1","addr":"172.21.15.8:6815","nonce":2761200318}]},"public_addr":"172.21.15.8:6811/2761200318","cluster_addr":"172.21.15.8:6813/2761200318","heartbeat_back_addr":"172.21.15.8:6817/2761200318","heartbeat_front_addr":"172.21.15.8:6815/2761200318","state":["exists","up"]},{"osd":2,"uuid":"ee8d332d-b536-4dfe-b0a2-f43d79149677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":3582626304},{"type":"v1","addr":"172.21.15.8:6819","nonce":3582626304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":3582626304},{"type":"v1","addr":"172.21.15.8:6821","nonce":3582626304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":3582626304},{"type":"v1","addr":"172.21.15.8:6825","nonce":3582626304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":3582626304},{"type":"v1","addr":"172.21.15.8:6823","nonce":3582626304}]},"public_addr":"172.21.15.8:6819/3582626304","cluster_addr":"172.21.15.8:6821/3582626304","heartbeat_back_addr":"172.21.15.8:6825/3582626304","heartbeat_front_addr":"172.21.15.8:6823/3582626304","state":["exists","up"]},{"osd":3,"uuid":"65bead61-5127-4144-8704-41a61d19aac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":2354457054},{"type":"v1","addr":"172.21.15.8:6827","nonce":2354457054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":2354457054},{"type":"v1","addr":"172.21.15.8:6829","nonce":2354457054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6832","nonce":2354457054},{"type":"v1","addr":"172.21.15.8:6833","nonce":2354457054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":2354457054},{"type":"v1","addr":"172.21.15.8:6831","nonce":2354457054}]},"public_addr":"172.21.15.8:6827/2354457054","cluster_addr":"172.21.15.8:6829/2354457054","heartbeat_back_addr":"172.21.15.8:6833/2354457054","heartbeat_front_addr":"172.21.15.8:6831/2354457054","state":["exists","up"]},{"osd":4,"uuid":"e81bc030-196f-4623-9d12-2bc124c12c1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2473086748},{"type":"v1","addr":"172.21.15.12:6801","nonce":2473086748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2473086748},{"type":"v1","addr":"172.21.15.12:6803","nonce":2473086748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2473086748},{"type":"v1","addr":"172.21.15.12:6807","nonce":2473086748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2473086748},{"type":"v1","addr":"172.21.15.12:6805","nonce":2473086748}]},"public_addr":"172.21.15.12:6801/2473086748","cluster_addr":"172.21.15.12:6803/2473086748","heartbeat_back_addr":"172.21.15.12:6807/2473086748","heartbeat_front_addr":"172.21.15.12:6805/2473086748","state":["exists","up"]},{"osd":5,"uuid":"a59bc6f0-504a-4c62-8164-2bf98525712a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3659656891},{"type":"v1","addr":"172.21.15.12:6809","nonce":3659656891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3659656891},{"type":"v1","addr":"172.21.15.12:6811","nonce":3659656891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3659656891},{"type":"v1","addr":"172.21.15.12:6815","nonce":3659656891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3659656891},{"type":"v1","addr":"172.21.15.12:6813","nonce":3659656891}]},"public_addr":"172.21.15.12:6809/3659656891","cluster_addr":"172.21.15.12:6811/3659656891","heartbeat_back_addr":"172.21.15.12:6815/3659656891","heartbeat_front_addr":"172.21.15.12:6813/3659656891","state":["exists","up"]},{"osd":6,"uuid":"6267d07e-3bea-4323-bc72-0e6bc68f3469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2575290587},{"type":"v1","addr":"172.21.15.12:6817","nonce":2575290587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":2575290587},{"type":"v1","addr":"172.21.15.12:6819","nonce":2575290587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":2575290587},{"type":"v1","addr":"172.21.15.12:6823","nonce":2575290587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":2575290587},{"type":"v1","addr":"172.21.15.12:6821","nonce":2575290587}]},"public_addr":"172.21.15.12:6817/2575290587","cluster_addr":"172.21.15.12:6819/2575290587","heartbeat_back_addr":"172.21.15.12:6823/2575290587","heartbeat_front_addr":"172.21.15.12:6821/2575290587","state":["exists","up"]},{"osd":7,"uuid":"3c59bc81-f988-4f95-96f0-be7924acb6df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3762015288},{"type":"v1","addr":"172.21.15.12:6825","nonce":3762015288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3762015288},{"type":"v1","addr":"172.21.15.12:6827","nonce":3762015288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3762015288},{"type":"v1","addr":"172.21.15.12:6831","nonce":3762015288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3762015288},{"type":"v1","addr":"172.21.15.12:6829","nonce":3762015288}]},"public_addr":"172.21.15.12:6825/3762015288","cluster_addr":"172.21.15.12:6827/3762015288","heartbeat_back_addr":"172.21.15.12:6831/3762015288","heartbeat_front_addr":"172.21.15.12:6829/3762015288","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:30:38.696430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:31:07.057278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:31:32.330617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:32:00.215681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:32:21.755098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:32:45.772849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:33:09.901851+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:33:35.261230+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.8:0/4061795296":"2024-04-15T13:29:31.790648+0000","172.21.15.8:6800/1546778513":"2024-04-15T13:29:31.790648+0000","172.21.15.8:0/3432677301":"2024-04-15T13:28:35.306236+0000","172.21.15.8:6801/1192676720":"2024-04-15T13:28:35.306236+0000","172.21.15.8:6800/1192676720":"2024-04-15T13:28:35.306236+0000","172.21.15.8:0/3050160566":"2024-04-15T13:29:31.790648+0000","172.21.15.8:0/1865709161":"2024-04-15T13:28:35.306236+0000","172.21.15.8:0/4035696983":"2024-04-15T13:28:35.306236+0000","172.21.15.8:6800/596187414":"2024-04-15T13:28:05.823103+0000","172.21.15.8:6801/596187414":"2024-04-15T13:28:05.823103+0000","172.21.15.8:0/1824349976":"2024-04-15T13:28:05.823103+0000","172.21.15.8:0/1159808486":"2024-04-15T13:29:31.790648+0000","172.21.15.8:0/2433183361":"2024-04-15T13:28:05.823103+0000","172.21.15.8:6801/1546778513":"2024-04-15T13:29:31.790648+0000","172.21.15.8:0/1505809637":"2024-04-15T13:28:05.823103+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T13:33:54.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:53 smithi012 ceph-mon[27236]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s, 0 objects/s recovering 2024-04-14T13:33:54.492 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-14T13:33:54.493 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-14T13:33:54.493 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-14T13:33:54.494 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-14T13:33:54.494 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-14T13:33:54.495 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-14T13:33:54.495 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-14T13:33:54.496 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-14T13:33:54.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:54 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/825795867' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T13:33:54.946 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:33:54.946 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:33:54.947 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:33:54.947 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:33:54.948 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:33:54.950 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:33:54.950 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:33:54.956 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:33:55.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:54 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/825795867' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T13:33:55.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:55 smithi008 ceph-mon[24786]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:33:56.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:55 smithi012 ceph-mon[27236]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:33:57.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:57 smithi008 ceph-mon[24786]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:33:58.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:57 smithi012 ceph-mon[27236]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:33:59.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:33:59 smithi008 ceph-mon[24786]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:00.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:33:59 smithi012 ceph-mon[27236]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:01.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:01 smithi008 ceph-mon[24786]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:02.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:01 smithi012 ceph-mon[27236]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:03.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:03 smithi008 ceph-mon[24786]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:04.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:03 smithi012 ceph-mon[27236]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:04.826 INFO:teuthology.orchestra.run.smithi008.stdout:111669149723 2024-04-14T13:34:04.826 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-14T13:34:05.849 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:05 smithi008 ceph-mon[24786]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:06.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:05 smithi012 ceph-mon[27236]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:06.718 INFO:teuthology.orchestra.run.smithi008.stdout:81604378658 2024-04-14T13:34:06.719 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-14T13:34:07.627 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:07 smithi008 ceph-mon[24786]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:08.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:07 smithi012 ceph-mon[27236]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:08.365 INFO:teuthology.orchestra.run.smithi008.stdout:38654705708 2024-04-14T13:34:08.365 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-14T13:34:09.535 INFO:teuthology.orchestra.run.smithi008.stdout:167503724561 2024-04-14T13:34:09.536 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-14T13:34:09.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:09 smithi008 ceph-mon[24786]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:10.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:09 smithi012 ceph-mon[27236]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:10.143 INFO:teuthology.orchestra.run.smithi008.stdout:201863462925 2024-04-14T13:34:10.143 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-14T13:34:10.647 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:34:10.648 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:34:10.648 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:34:11.934 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:11 smithi008 ceph-mon[24786]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:12.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:11 smithi012 ceph-mon[27236]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:12.050 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:34:12.051 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:34:12.815 INFO:teuthology.orchestra.run.smithi008.stdout:60129542184 2024-04-14T13:34:12.815 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-14T13:34:13.947 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:13 smithi008 ceph-mon[24786]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:14.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:13 smithi012 ceph-mon[27236]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:15.038 INFO:teuthology.orchestra.run.smithi008.stdout:133143986200 2024-04-14T13:34:15.038 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-14T13:34:15.427 INFO:teuthology.orchestra.run.smithi008.stdout:236223201289 2024-04-14T13:34:15.427 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-14T13:34:15.652 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:15 smithi008 ceph-mon[24786]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:15.820 INFO:teuthology.orchestra.run.smithi008.stdout:81604378659 2024-04-14T13:34:16.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:15 smithi012 ceph-mon[27236]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:16.482 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378658 got 81604378659 for osd.2 2024-04-14T13:34:16.482 DEBUG:teuthology.parallel:result is None 2024-04-14T13:34:16.745 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:16 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/605294819' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-14T13:34:17.040 INFO:teuthology.orchestra.run.smithi008.stdout:38654705709 2024-04-14T13:34:17.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:16 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/605294819' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-14T13:34:17.741 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:17 smithi008 ceph-mon[24786]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:17.742 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:17 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2581602971' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-14T13:34:18.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:17 smithi012 ceph-mon[27236]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:18.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:17 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/2581602971' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-14T13:34:18.192 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:34:18.203 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:34:18.217 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:34:19.125 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705708 got 38654705709 for osd.0 2024-04-14T13:34:19.126 DEBUG:teuthology.parallel:result is None 2024-04-14T13:34:19.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:19 smithi008 ceph-mon[24786]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:20.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:19 smithi012 ceph-mon[27236]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:20.895 INFO:teuthology.orchestra.run.smithi008.stdout:111669149726 2024-04-14T13:34:20.895 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:20 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2731867518' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-14T13:34:21.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:20 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/2731867518' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-14T13:34:21.448 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149723 got 111669149726 for osd.3 2024-04-14T13:34:21.448 DEBUG:teuthology.parallel:result is None 2024-04-14T13:34:21.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:21 smithi008 ceph-mon[24786]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:22.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:21 smithi012 ceph-mon[27236]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:23.974 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:23 smithi008 ceph-mon[24786]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:24.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:23 smithi012 ceph-mon[27236]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:24.648 INFO:teuthology.orchestra.run.smithi008.stdout:167503724566 2024-04-14T13:34:25.227 INFO:teuthology.orchestra.run.smithi008.stdout:133143986203 2024-04-14T13:34:25.318 INFO:teuthology.orchestra.run.smithi008.stdout:201863462929 2024-04-14T13:34:26.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:25 smithi012 ceph-mon[27236]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:26.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:25 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/659379410' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-14T13:34:26.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:25 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/2782429412' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-14T13:34:26.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:25 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/2530999039' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-14T13:34:26.070 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:25 smithi008 ceph-mon[24786]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:26.070 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:25 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/659379410' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-14T13:34:26.070 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:25 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2782429412' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-14T13:34:26.070 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:25 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2530999039' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-14T13:34:26.093 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462925 got 201863462929 for osd.6 2024-04-14T13:34:26.093 DEBUG:teuthology.parallel:result is None 2024-04-14T13:34:26.592 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724561 got 167503724566 for osd.5 2024-04-14T13:34:26.623 DEBUG:teuthology.parallel:result is None 2024-04-14T13:34:27.177 INFO:teuthology.orchestra.run.smithi008.stdout:236223201292 2024-04-14T13:34:27.326 INFO:teuthology.orchestra.run.smithi008.stdout:60129542187 2024-04-14T13:34:27.529 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986200 got 133143986203 for osd.4 2024-04-14T13:34:27.529 DEBUG:teuthology.parallel:result is None 2024-04-14T13:34:27.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:27 smithi008 ceph-mon[24786]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:27.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:27 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/73644424' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-14T13:34:27.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:27 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/4049661614' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-14T13:34:28.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:27 smithi012 ceph-mon[27236]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:28.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:27 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/73644424' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-14T13:34:28.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:27 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/4049661614' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-14T13:34:28.153 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542184 got 60129542187 for osd.1 2024-04-14T13:34:28.153 DEBUG:teuthology.parallel:result is None 2024-04-14T13:34:28.791 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201289 got 236223201292 for osd.7 2024-04-14T13:34:28.791 DEBUG:teuthology.parallel:result is None 2024-04-14T13:34:28.792 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-14T13:34:28.792 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T13:34:29.020 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:34:29.803 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:29 smithi008 ceph-mon[24786]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:30.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:29 smithi012 ceph-mon[27236]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:30.694 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:34:30.695 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-04-14T13:34:31.317 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-04-14T13:34:29.895647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48816,"kb_used_data":6320,"kb_used_omap":0,"kb_used_meta":42432,"kb_avail":749912400,"statfs":{"total":767960285184,"available":767910297600,"internally_reserved":0,"allocated":6471680,"data_stored":3548060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43450368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002178"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":226,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-14T13:33:40.563799+0000","last_change":"2024-04-14T13:33:40.563799+0000","last_active":"2024-04-14T13:33:40.563799+0000","last_peered":"2024-04-14T13:33:40.563799+0000","last_clean":"2024-04-14T13:33:40.563799+0000","last_became_active":"2024-04-14T13:33:40.563417+0000","last_became_peered":"2024-04-14T13:33:40.563417+0000","last_unstale":"2024-04-14T13:33:40.563799+0000","last_undegraded":"2024-04-14T13:33:40.563799+0000","last_fullsized":"2024-04-14T13:33:40.563799+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T13:31:34.759031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T13:31:34.759031+0000","last_clean_scrub_stamp":"2024-04-14T13:31:34.759031+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:41:59.184262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201293,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6052,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739100,"statfs":{"total":95995035648,"available":95988838400,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]}]},{"osd":6,"up_from":47,"seq":201863462930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6176,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738976,"statfs":{"total":95995035648,"available":95988711424,"internally_reserved":0,"allocated":1138688,"data_stored":773838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.30299999999999999}]},{"osd":1,"last update":"Sun Apr 14 13:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.52100000000000002}]},{"osd":2,"last update":"Sun Apr 14 13:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.38200000000000001}]},{"osd":3,"last update":"Sun Apr 14 13:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.55000000000000004}]},{"osd":4,"last update":"Sun Apr 14 13:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.439}]},{"osd":5,"last update":"Sun Apr 14 13:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.53000000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6364,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.35899999999999999,"15min":0.33800000000000002},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.56399999999999995,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.312},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.377,"15min":0.35699999999999998},"min":{"1min":0.245,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.621,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.34399999999999997}]},{"osd":2,"last update":"Sun Apr 14 13:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.39800000000000002,"15min":0.39000000000000001},"min":{"1min":0.316,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.42599999999999999,"15min":0.41599999999999998},"min":{"1min":0.34699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.36199999999999999}]},{"osd":3,"last update":"Sun Apr 14 13:34:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.44900000000000001,"15min":0.44800000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.45100000000000001}]},{"osd":4,"last update":"Sun Apr 14 13:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.61599999999999999,"15min":0.61499999999999999},"min":{"1min":0.46800000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.78100000000000003,"5min":1.028,"15min":1.028},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60199999999999998,"15min":0.60599999999999998},"min":{"1min":0.45200000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.81999999999999995,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.622}]},{"osd":5,"last update":"Sun Apr 14 13:33:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.505}]},{"osd":6,"last update":"Sun Apr 14 13:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.53500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705712,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Apr 14 13:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.34000000000000002,"15min":0.33000000000000002},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.36099999999999999,"15min":0.33900000000000002},"min":{"1min":0.25700000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.372}]},{"osd":2,"last update":"Sun Apr 14 13:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.373,"15min":0.36899999999999999},"min":{"1min":0.29499999999999998,"5min":0.183,"15min":0.183},"max":{"1min":0.53800000000000003,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.39300000000000002,"15min":0.38500000000000001},"min":{"1min":0.29099999999999998,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.54700000000000004,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.35899999999999999}]},{"osd":3,"last update":"Sun Apr 14 13:34:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.433,"15min":0.437},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45900000000000002,"15min":0.46000000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.39400000000000002}]},{"osd":4,"last update":"Sun Apr 14 13:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.71899999999999997}]},{"osd":5,"last update":"Sun Apr 14 13:33:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.74299999999999999}]},{"osd":6,"last update":"Sun Apr 14 13:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.77900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378663,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6236,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738916,"statfs":{"total":95995035648,"available":95988649984,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.35499999999999998,"15min":0.34699999999999998},"min":{"1min":0.28599999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.39400000000000002,"15min":0.38},"min":{"1min":0.34399999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.32200000000000001}]},{"osd":1,"last update":"Sun Apr 14 13:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.38100000000000001,"15min":0.375},"min":{"1min":0.30499999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.40300000000000002,"15min":0.39300000000000002},"min":{"1min":0.317,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.46700000000000003}]},{"osd":3,"last update":"Sun Apr 14 13:34:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.46999999999999997,"15min":0.47299999999999998},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.70799999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.47099999999999997,"15min":0.47799999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.52500000000000002}]},{"osd":4,"last update":"Sun Apr 14 13:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.59599999999999997,"15min":0.59399999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.46700000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.505}]},{"osd":5,"last update":"Sun Apr 14 13:33:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.56399999999999995}]},{"osd":6,"last update":"Sun Apr 14 13:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.754},{"interface":"front","average":{"1min":1.49,"5min":1.49,"15min":1.49},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":11.855,"5min":11.855,"15min":11.855},"last":0.55600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.38900000000000001,"15min":0.38100000000000001},"min":{"1min":0.25600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.39900000000000002,"15min":0.39300000000000002},"min":{"1min":0.27000000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.36199999999999999}]},{"osd":1,"last update":"Sun Apr 14 13:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.41999999999999998,"15min":0.41799999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.41399999999999998,"15min":0.40600000000000003},"min":{"1min":0.31,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.32900000000000001}]},{"osd":2,"last update":"Sun Apr 14 13:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.42199999999999999,"15min":0.41399999999999998},"min":{"1min":0.32200000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.376},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.41299999999999998,"15min":0.40699999999999997},"min":{"1min":0.28399999999999997,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.34499999999999997}]},{"osd":4,"last update":"Sun Apr 14 13:33:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.53900000000000003}]},{"osd":5,"last update":"Sun Apr 14 13:33:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.47999999999999998}]},{"osd":6,"last update":"Sun Apr 14 13:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.64800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986205,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5720,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739432,"statfs":{"total":95995035648,"available":95989178368,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57799999999999996,"15min":0.58399999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.74399999999999999,"5min":0.752,"15min":0.752},"last":0.56999999999999995}]},{"osd":1,"last update":"Sun Apr 14 13:34:26 2024","interfaces":[{"interface":"back","average":{"1min":1.7430000000000001,"5min":0.82299999999999995,"15min":0.66900000000000004},"min":{"1min":0.40999999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":19.161999999999999,"5min":19.161999999999999,"15min":19.161999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.39500000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.63500000000000001}]},{"osd":2,"last update":"Sun Apr 14 13:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57099999999999995,"15min":0.56899999999999995},"min":{"1min":0.44900000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.59099999999999997,"15min":0.59599999999999997},"min":{"1min":0.38800000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.70399999999999996,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.53500000000000003}]},{"osd":3,"last update":"Sun Apr 14 13:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.38900000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57799999999999996,"15min":0.57899999999999996},"min":{"1min":0.40200000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.745,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.56399999999999995}]},{"osd":5,"last update":"Sun Apr 14 13:33:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.40000000000000002}]},{"osd":6,"last update":"Sun Apr 14 13:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.504}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]}]},{"osd":5,"up_from":39,"seq":167503724567,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5660,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739492,"statfs":{"total":95995035648,"available":95989239808,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:33:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.56499999999999995}]},{"osd":1,"last update":"Sun Apr 14 13:33:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.435},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.63700000000000001}]},{"osd":2,"last update":"Sun Apr 14 13:33:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.499},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.58099999999999996}]},{"osd":3,"last update":"Sun Apr 14 13:33:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.50800000000000001}]},{"osd":4,"last update":"Sun Apr 14 13:33:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.70699999999999996}]},{"osd":6,"last update":"Sun Apr 14 13:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.66900000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T13:34:31.318 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T13:34:31.536 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:34:31.979 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:31 smithi008 ceph-mon[24786]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:31.979 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:31 smithi008 ceph-mon[24786]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:34:32.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:31 smithi012 ceph-mon[27236]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:32.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:31 smithi012 ceph-mon[27236]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:34:32.976 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:34:32.976 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:34:33.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:34:33.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:34:33.359 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:34:33.362 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-04-14T13:34:33.940 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:33 smithi008 ceph-mon[24786]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:33.966 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-04-14T13:34:31.896102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48816,"kb_used_data":6320,"kb_used_omap":0,"kb_used_meta":42432,"kb_avail":749912400,"statfs":{"total":767960285184,"available":767910297600,"internally_reserved":0,"allocated":6471680,"data_stored":3548060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43450368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002309"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":226,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-14T13:33:40.563799+0000","last_change":"2024-04-14T13:33:40.563799+0000","last_active":"2024-04-14T13:33:40.563799+0000","last_peered":"2024-04-14T13:33:40.563799+0000","last_clean":"2024-04-14T13:33:40.563799+0000","last_became_active":"2024-04-14T13:33:40.563417+0000","last_became_peered":"2024-04-14T13:33:40.563417+0000","last_unstale":"2024-04-14T13:33:40.563799+0000","last_undegraded":"2024-04-14T13:33:40.563799+0000","last_fullsized":"2024-04-14T13:33:40.563799+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T13:31:34.759031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T13:31:34.759031+0000","last_clean_scrub_stamp":"2024-04-14T13:31:34.759031+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:41:59.184262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201294,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6052,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739100,"statfs":{"total":95995035648,"available":95988838400,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]}]},{"osd":6,"up_from":47,"seq":201863462931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6176,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738976,"statfs":{"total":95995035648,"available":95988711424,"internally_reserved":0,"allocated":1138688,"data_stored":773838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.69799999999999995}]},{"osd":1,"last update":"Sun Apr 14 13:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.57799999999999996}]},{"osd":2,"last update":"Sun Apr 14 13:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.73799999999999999}]},{"osd":3,"last update":"Sun Apr 14 13:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.70999999999999996}]},{"osd":4,"last update":"Sun Apr 14 13:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.55500000000000005}]},{"osd":5,"last update":"Sun Apr 14 13:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.59999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6364,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.35899999999999999,"15min":0.33800000000000002},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.56399999999999995,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.312},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.377,"15min":0.35699999999999998},"min":{"1min":0.245,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.621,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.34399999999999997}]},{"osd":2,"last update":"Sun Apr 14 13:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.39800000000000002,"15min":0.39000000000000001},"min":{"1min":0.316,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.42599999999999999,"15min":0.41599999999999998},"min":{"1min":0.34699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.36199999999999999}]},{"osd":3,"last update":"Sun Apr 14 13:34:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.44900000000000001,"15min":0.44800000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.45100000000000001}]},{"osd":4,"last update":"Sun Apr 14 13:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.61599999999999999,"15min":0.61499999999999999},"min":{"1min":0.46800000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.78100000000000003,"5min":1.028,"15min":1.028},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60199999999999998,"15min":0.60599999999999998},"min":{"1min":0.45200000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.81999999999999995,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.622}]},{"osd":5,"last update":"Sun Apr 14 13:33:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.505}]},{"osd":6,"last update":"Sun Apr 14 13:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.53500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Apr 14 13:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.34000000000000002,"15min":0.33000000000000002},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.36099999999999999,"15min":0.33900000000000002},"min":{"1min":0.25700000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.39300000000000002}]},{"osd":2,"last update":"Sun Apr 14 13:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.373,"15min":0.36899999999999999},"min":{"1min":0.29499999999999998,"5min":0.183,"15min":0.183},"max":{"1min":0.53800000000000003,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.378},{"interface":"front","average":{"1min":0.438,"5min":0.39300000000000002,"15min":0.38500000000000001},"min":{"1min":0.29099999999999998,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.54700000000000004,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.28299999999999997}]},{"osd":3,"last update":"Sun Apr 14 13:34:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.433,"15min":0.437},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45900000000000002,"15min":0.46000000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.35699999999999998}]},{"osd":4,"last update":"Sun Apr 14 13:34:29 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60099999999999998,"15min":0.59699999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.61499999999999999,"15min":0.61299999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.40899999999999997}]},{"osd":5,"last update":"Sun Apr 14 13:33:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.44},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.50900000000000001}]},{"osd":6,"last update":"Sun Apr 14 13:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.49299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378663,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6236,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738916,"statfs":{"total":95995035648,"available":95988649984,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.35499999999999998,"15min":0.34699999999999998},"min":{"1min":0.28599999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.39400000000000002,"15min":0.38},"min":{"1min":0.34399999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.32200000000000001}]},{"osd":1,"last update":"Sun Apr 14 13:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.38100000000000001,"15min":0.375},"min":{"1min":0.30499999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.40300000000000002,"15min":0.39300000000000002},"min":{"1min":0.317,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.46700000000000003}]},{"osd":3,"last update":"Sun Apr 14 13:34:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.46999999999999997,"15min":0.47299999999999998},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.70799999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.47099999999999997,"15min":0.47799999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.52500000000000002}]},{"osd":4,"last update":"Sun Apr 14 13:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.59599999999999997,"15min":0.59399999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.46700000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.505}]},{"osd":5,"last update":"Sun Apr 14 13:33:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.56399999999999995}]},{"osd":6,"last update":"Sun Apr 14 13:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.754},{"interface":"front","average":{"1min":1.49,"5min":1.49,"15min":1.49},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":11.855,"5min":11.855,"15min":11.855},"last":0.55600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.38900000000000001,"15min":0.38100000000000001},"min":{"1min":0.25600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.39900000000000002,"15min":0.39300000000000002},"min":{"1min":0.27000000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.36199999999999999}]},{"osd":1,"last update":"Sun Apr 14 13:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.41999999999999998,"15min":0.41799999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.41399999999999998,"15min":0.40600000000000003},"min":{"1min":0.31,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.32900000000000001}]},{"osd":2,"last update":"Sun Apr 14 13:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.42199999999999999,"15min":0.41399999999999998},"min":{"1min":0.32200000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.376},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.41299999999999998,"15min":0.40699999999999997},"min":{"1min":0.28399999999999997,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.34499999999999997}]},{"osd":4,"last update":"Sun Apr 14 13:33:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.53900000000000003}]},{"osd":5,"last update":"Sun Apr 14 13:33:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.47999999999999998}]},{"osd":6,"last update":"Sun Apr 14 13:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.64800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986205,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5720,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739432,"statfs":{"total":95995035648,"available":95989178368,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57799999999999996,"15min":0.58399999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.74399999999999999,"5min":0.752,"15min":0.752},"last":0.56999999999999995}]},{"osd":1,"last update":"Sun Apr 14 13:34:26 2024","interfaces":[{"interface":"back","average":{"1min":1.7430000000000001,"5min":0.82299999999999995,"15min":0.66900000000000004},"min":{"1min":0.40999999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":19.161999999999999,"5min":19.161999999999999,"15min":19.161999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.39500000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.63500000000000001}]},{"osd":2,"last update":"Sun Apr 14 13:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57099999999999995,"15min":0.56899999999999995},"min":{"1min":0.44900000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.59099999999999997,"15min":0.59599999999999997},"min":{"1min":0.38800000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.70399999999999996,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.53500000000000003}]},{"osd":3,"last update":"Sun Apr 14 13:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.38900000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57799999999999996,"15min":0.57899999999999996},"min":{"1min":0.40200000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.745,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.56399999999999995}]},{"osd":5,"last update":"Sun Apr 14 13:33:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.40000000000000002}]},{"osd":6,"last update":"Sun Apr 14 13:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.504}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]}]},{"osd":5,"up_from":39,"seq":167503724567,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5660,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739492,"statfs":{"total":95995035648,"available":95989239808,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:33:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.56499999999999995}]},{"osd":1,"last update":"Sun Apr 14 13:33:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.435},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.63700000000000001}]},{"osd":2,"last update":"Sun Apr 14 13:33:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.499},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.58099999999999996}]},{"osd":3,"last update":"Sun Apr 14 13:33:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.50800000000000001}]},{"osd":4,"last update":"Sun Apr 14 13:33:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.70699999999999996}]},{"osd":6,"last update":"Sun Apr 14 13:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.66900000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T13:34:33.967 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-14T13:34:33.967 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-14T13:34:33.967 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-14T13:34:33.967 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph health --format=json 2024-04-14T13:34:34.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:33 smithi012 ceph-mon[27236]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:34.188 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/mon.smithi008/config 2024-04-14T13:34:34.954 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:34 smithi008 ceph-mon[24786]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:34:35.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:34 smithi012 ceph-mon[27236]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:34:35.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:35 smithi008 ceph-mon[24786]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:35.942 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:34:35.943 INFO:teuthology.orchestra.run.smithi008.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-14T13:34:36.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:35 smithi012 ceph-mon[27236]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:36.582 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-14T13:34:36.582 INFO:tasks.cephadm:Setup complete, yielding 2024-04-14T13:34:36.583 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-14T13:34:36.592 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-14T13:34:36.593 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- bash -c 'ceph orch status' 2024-04-14T13:34:36.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:36 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/3585249392' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-14T13:34:37.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:36 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/3585249392' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-14T13:34:37.809 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:37 smithi008 ceph-mon[24786]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:38.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:37 smithi012 ceph-mon[27236]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:38.471 INFO:teuthology.orchestra.run.smithi008.stdout:Backend: cephadm 2024-04-14T13:34:38.472 INFO:teuthology.orchestra.run.smithi008.stdout:Available: Yes 2024-04-14T13:34:38.472 INFO:teuthology.orchestra.run.smithi008.stdout:Paused: No 2024-04-14T13:34:39.084 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-14T13:34:39.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:39 smithi012 ceph-mon[27236]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:39 smithi012 ceph-mon[27236]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:34:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:39 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:34:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:39 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:34:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:39 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:34:39.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:39 smithi008 ceph-mon[24786]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:39.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:39 smithi008 ceph-mon[24786]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:34:39.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:34:39.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:34:39.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:39 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:34:40.970 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:34:40.970 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (4m) 2m ago 5m 14.1M - ba2b418f427c fbc215a6ff5d 2024-04-14T13:34:40.970 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (5m) 2m ago 5m 6405k - 17.2.0 e1d6a67b021e 68b810a9ff93 2024-04-14T13:34:40.971 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (4m) 64s ago 4m 6400k - 17.2.0 e1d6a67b021e 6bc31c51b938 2024-04-14T13:34:40.971 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (3m) 2m ago 5m 43.2M - 8.3.5 dad864ee21e9 c7b42d05cedc 2024-04-14T13:34:40.971 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:9283 running (6m) 2m ago 6m 448M - 17.2.0 e1d6a67b021e 074f98e540c3 2024-04-14T13:34:40.971 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283 running (4m) 64s ago 4m 411M - 17.2.0 e1d6a67b021e 596e23b54c3d 2024-04-14T13:34:40.971 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (7m) 2m ago 7m 44.7M 2048M 17.2.0 e1d6a67b021e 7786c27a60d0 2024-04-14T13:34:40.971 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (4m) 64s ago 4m 39.8M 2048M 17.2.0 e1d6a67b021e f6cd483dfa31 2024-04-14T13:34:40.971 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (5m) 2m ago 5m 16.7M - 1dbe0e931976 2338da898186 2024-04-14T13:34:40.971 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (4m) 64s ago 4m 16.6M - 1dbe0e931976 c2fae00595dd 2024-04-14T13:34:40.971 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (4m) 2m ago 4m 33.9M 3217M 17.2.0 e1d6a67b021e 3b8cdea7890c 2024-04-14T13:34:40.971 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (3m) 2m ago 3m 35.4M 3217M 17.2.0 e1d6a67b021e be86642c7790 2024-04-14T13:34:40.971 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (3m) 2m ago 3m 33.6M 3217M 17.2.0 e1d6a67b021e b1017f00578f 2024-04-14T13:34:40.971 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (2m) 2m ago 2m 27.8M 3217M 17.2.0 e1d6a67b021e 66fca4b191be 2024-04-14T13:34:40.971 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (2m) 64s ago 2m 34.7M 3985M 17.2.0 e1d6a67b021e 43b02cb64951 2024-04-14T13:34:40.971 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (119s) 64s ago 119s 32.4M 3985M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:34:40.971 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (95s) 64s ago 95s 30.5M 3985M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:34:40.971 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (70s) 64s ago 69s 13.1M 3985M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:34:40.972 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (4m) 2m ago 4m 55.8M - 514e6a882f6e 11fca64796ed 2024-04-14T13:34:41.263 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:41 smithi008 ceph-mon[24786]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:41.263 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:41 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:34:41.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:41 smithi012 ceph-mon[27236]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:41.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:41 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:34:41.581 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-14T13:34:42.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:42 smithi008 ceph-mon[24786]: from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:34:42.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:42 smithi012 ceph-mon[27236]: from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:34:43.154 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:43 smithi008 ceph-mon[24786]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:43.547 INFO:teuthology.orchestra.run.smithi008.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-14T13:34:43.547 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-14T13:34:43.547 INFO:teuthology.orchestra.run.smithi008.stdout:crash 2/2 2m ago 6m * 2024-04-14T13:34:43.547 INFO:teuthology.orchestra.run.smithi008.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-14T13:34:43.547 INFO:teuthology.orchestra.run.smithi008.stdout:mgr 2/2 2m ago 6m count:2 2024-04-14T13:34:43.547 INFO:teuthology.orchestra.run.smithi008.stdout:mon 2/2 2m ago 5m smithi008:172.21.15.8=smithi008;smithi012:172.21.15.12=smithi012;count:2 2024-04-14T13:34:43.547 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-14T13:34:43.547 INFO:teuthology.orchestra.run.smithi008.stdout:osd 8 2m ago - 2024-04-14T13:34:43.548 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-04-14T13:34:43.549 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:43 smithi012 ceph-mon[27236]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:44.179 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-14T13:34:44.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:44 smithi008 ceph-mon[24786]: from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:34:44.549 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:44 smithi012 ceph-mon[27236]: from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:34:45.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:45 smithi008 ceph-mon[24786]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:45.451 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:45 smithi012 ceph-mon[27236]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:46.092 INFO:teuthology.orchestra.run.smithi008.stdout:HOST ADDR LABELS STATUS 2024-04-14T13:34:46.092 INFO:teuthology.orchestra.run.smithi008.stdout:smithi008 172.21.15.8 2024-04-14T13:34:46.092 INFO:teuthology.orchestra.run.smithi008.stdout:smithi012 172.21.15.12 2024-04-14T13:34:46.092 INFO:teuthology.orchestra.run.smithi008.stdout:2 hosts in cluster 2024-04-14T13:34:46.690 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-14T13:34:47.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:47 smithi008 ceph-mon[24786]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:47.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:47 smithi008 ceph-mon[24786]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:34:47.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:47 smithi012 ceph-mon[27236]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:47.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:47 smithi012 ceph-mon[27236]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:34:48.635 INFO:teuthology.orchestra.run.smithi008.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2024-04-14T13:34:48.636 INFO:teuthology.orchestra.run.smithi008.stdout:smithi008 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476004A480DGN 480G LVM detected, locked 2024-04-14T13:34:48.636 INFO:teuthology.orchestra.run.smithi008.stdout:smithi012 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331005J400BGN 400G LVM detected, locked 2024-04-14T13:34:49.210 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-14T13:34:49.222 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-14T13:34:49.222 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-04-14T13:34:49.249 INFO:teuthology.orchestra.run.smithi008.stderr:+ systemctl stop nfs-server 2024-04-14T13:34:49.256 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi012.front.sepia.ceph.com 2024-04-14T13:34:49.256 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-04-14T13:34:49.289 INFO:teuthology.orchestra.run.smithi012.stderr:+ systemctl stop nfs-server 2024-04-14T13:34:49.297 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-14T13:34:49.305 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-14T13:34:49.305 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- bash -c 'ceph fs volume create foofs' 2024-04-14T13:34:49.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:49 smithi012 ceph-mon[27236]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:49.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:49 smithi012 ceph-mon[27236]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:34:49.838 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:49 smithi008 ceph-mon[24786]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:49.838 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:49 smithi008 ceph-mon[24786]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:34:51.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:51 smithi012 ceph-mon[27236]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:51.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:51 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-04-14T13:34:51.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:51 smithi008 ceph-mon[24786]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:51.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:51 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-04-14T13:34:52.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:52 smithi008 ceph-mon[24786]: from='client.14498 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:34:52.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:52 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-04-14T13:34:52.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:52 smithi008 ceph-mon[24786]: osdmap e60: 8 total, 8 up, 8 in 2024-04-14T13:34:52.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:52 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-04-14T13:34:52.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:52 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008[24782]: 2024-04-14T13:34:52.552+0000 7f4c2146c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-14T13:34:53.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:52 smithi012 ceph-mon[27236]: from='client.14498 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:34:53.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:52 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-04-14T13:34:53.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:52 smithi012 ceph-mon[27236]: osdmap e60: 8 total, 8 up, 8 in 2024-04-14T13:34:53.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:52 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-04-14T13:34:53.222 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-04-14T13:34:53.232 INFO:tasks.cephadm:Waiting for ceph service mds.foofs to start (timeout 300)... 2024-04-14T13:34:53.232 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph orch ls -f json 2024-04-14T13:34:53.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: pgmap v208: 33 pgs: 13 creating+peering, 19 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:53.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-04-14T13:34:53.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: osdmap e61: 8 total, 8 up, 8 in 2024-04-14T13:34:53.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-04-14T13:34:53.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-14T13:34:53.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-14T13:34:53.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: osdmap e62: 8 total, 8 up, 8 in 2024-04-14T13:34:53.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-04-14T13:34:53.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: fsmap foofs:0 2024-04-14T13:34:53.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: Saving service mds.foofs spec with placement count:2 2024-04-14T13:34:53.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:34:53.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:34:53.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:34:53.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:34:53.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:34:53.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:34:53.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.wqbvfi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-14T13:34:53.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.wqbvfi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-14T13:34:53.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:34:53.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:53 smithi012 ceph-mon[27236]: Deploying daemon mds.foofs.smithi008.wqbvfi on smithi008 2024-04-14T13:34:53.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: pgmap v208: 33 pgs: 13 creating+peering, 19 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:53.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-04-14T13:34:53.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: osdmap e61: 8 total, 8 up, 8 in 2024-04-14T13:34:53.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-04-14T13:34:53.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-14T13:34:53.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-14T13:34:53.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: osdmap e62: 8 total, 8 up, 8 in 2024-04-14T13:34:53.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-04-14T13:34:53.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: fsmap foofs:0 2024-04-14T13:34:53.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: Saving service mds.foofs spec with placement count:2 2024-04-14T13:34:53.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:34:53.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:34:53.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:34:53.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:34:53.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:34:53.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:34:53.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.wqbvfi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-14T13:34:53.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.wqbvfi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-14T13:34:53.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:34:53.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:53 smithi008 ceph-mon[24786]: Deploying daemon mds.foofs.smithi008.wqbvfi on smithi008 2024-04-14T13:34:54.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:54 smithi008 ceph-mon[24786]: osdmap e63: 8 total, 8 up, 8 in 2024-04-14T13:34:55.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:54 smithi012 ceph-mon[27236]: osdmap e63: 8 total, 8 up, 8 in 2024-04-14T13:34:55.761 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:55 smithi008 ceph-mon[24786]: pgmap v212: 65 pgs: 1 creating+activating, 19 creating+peering, 32 unknown, 13 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:55.761 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:55 smithi008 ceph-mon[24786]: osdmap e64: 8 total, 8 up, 8 in 2024-04-14T13:34:56.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:55 smithi012 ceph-mon[27236]: pgmap v212: 65 pgs: 1 creating+activating, 19 creating+peering, 32 unknown, 13 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:56.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:55 smithi012 ceph-mon[27236]: osdmap e64: 8 total, 8 up, 8 in 2024-04-14T13:34:56.181 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:34:56.182 INFO:teuthology.orchestra.run.smithi008.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-14T13:28:24.640841Z", "last_refresh": "2024-04-14T13:32:09.993071Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-14T13:28:19.222610Z", "last_refresh": "2024-04-14T13:32:09.993156Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-14T13:28:22.183609Z", "last_refresh": "2024-04-14T13:32:09.993226Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-14T13:34:52.574084Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-14T13:34:52.565217Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-14T13:28:17.214703Z", "last_refresh": "2024-04-14T13:32:09.992995Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi008:172.21.15.8=smithi008", "smithi012:172.21.15.12=smithi012"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-14T13:29:07.440988Z", "last_refresh": "2024-04-14T13:32:09.992849Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-14T13:28:23.319722Z", "last_refresh": "2024-04-14T13:32:09.993292Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-14T13:32:09.993421Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-14T13:28:20.882885Z", "last_refresh": "2024-04-14T13:32:09.993357Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-14T13:34:57.028 INFO:tasks.cephadm:mds.foofs has 0/2 2024-04-14T13:34:57.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:57 smithi008 ceph-mon[24786]: pgmap v214: 65 pgs: 1 creating+activating, 19 creating+peering, 14 unknown, 31 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:57.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:57 smithi008 ceph-mon[24786]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:34:57.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:57 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:34:57.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:57 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi012.vjovzp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-14T13:34:57.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:57 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi012.vjovzp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-14T13:34:57.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:57 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:34:57.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:57 smithi008 ceph-mon[24786]: Deploying daemon mds.foofs.smithi012.vjovzp on smithi012 2024-04-14T13:34:57.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:57 smithi008 ceph-mon[24786]: mds.? [v2:172.21.15.8:6834/785972446,v1:172.21.15.8:6835/785972446] up:boot 2024-04-14T13:34:57.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:57 smithi008 ceph-mon[24786]: daemon mds.foofs.smithi008.wqbvfi assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-04-14T13:34:57.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:57 smithi008 ceph-mon[24786]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-14T13:34:57.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:57 smithi008 ceph-mon[24786]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-14T13:34:57.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:57 smithi008 ceph-mon[24786]: Cluster is now healthy 2024-04-14T13:34:57.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:57 smithi008 ceph-mon[24786]: fsmap foofs:0 1 up:standby 2024-04-14T13:34:57.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:57 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wqbvfi"}]: dispatch 2024-04-14T13:34:57.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:57 smithi008 ceph-mon[24786]: fsmap foofs:1 {0=foofs.smithi008.wqbvfi=up:creating} 2024-04-14T13:34:57.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:57 smithi008 ceph-mon[24786]: daemon mds.foofs.smithi008.wqbvfi is now active in filesystem foofs as rank 0 2024-04-14T13:34:58.029 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph orch ls -f json 2024-04-14T13:34:58.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:57 smithi012 ceph-mon[27236]: pgmap v214: 65 pgs: 1 creating+activating, 19 creating+peering, 14 unknown, 31 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:57 smithi012 ceph-mon[27236]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:34:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:57 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:34:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:57 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi012.vjovzp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-14T13:34:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:57 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi012.vjovzp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-14T13:34:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:57 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:34:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:57 smithi012 ceph-mon[27236]: Deploying daemon mds.foofs.smithi012.vjovzp on smithi012 2024-04-14T13:34:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:57 smithi012 ceph-mon[27236]: mds.? [v2:172.21.15.8:6834/785972446,v1:172.21.15.8:6835/785972446] up:boot 2024-04-14T13:34:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:57 smithi012 ceph-mon[27236]: daemon mds.foofs.smithi008.wqbvfi assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-04-14T13:34:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:57 smithi012 ceph-mon[27236]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-14T13:34:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:57 smithi012 ceph-mon[27236]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-14T13:34:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:57 smithi012 ceph-mon[27236]: Cluster is now healthy 2024-04-14T13:34:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:57 smithi012 ceph-mon[27236]: fsmap foofs:0 1 up:standby 2024-04-14T13:34:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:57 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wqbvfi"}]: dispatch 2024-04-14T13:34:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:57 smithi012 ceph-mon[27236]: fsmap foofs:1 {0=foofs.smithi008.wqbvfi=up:creating} 2024-04-14T13:34:58.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:57 smithi012 ceph-mon[27236]: daemon mds.foofs.smithi008.wqbvfi is now active in filesystem foofs as rank 0 2024-04-14T13:34:59.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:59 smithi012 ceph-mon[27236]: pgmap v215: 65 pgs: 1 creating+activating, 6 creating+peering, 58 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:59.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:59 smithi012 ceph-mon[27236]: mds.? [v2:172.21.15.8:6834/785972446,v1:172.21.15.8:6835/785972446] up:active 2024-04-14T13:34:59.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:59 smithi012 ceph-mon[27236]: fsmap foofs:1 {0=foofs.smithi008.wqbvfi=up:active} 2024-04-14T13:34:59.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:59 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:34:59.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:59 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:34:59.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:59 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:34:59.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:34:59 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:34:59.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:59 smithi008 ceph-mon[24786]: pgmap v215: 65 pgs: 1 creating+activating, 6 creating+peering, 58 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:34:59.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:59 smithi008 ceph-mon[24786]: mds.? [v2:172.21.15.8:6834/785972446,v1:172.21.15.8:6835/785972446] up:active 2024-04-14T13:34:59.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:59 smithi008 ceph-mon[24786]: fsmap foofs:1 {0=foofs.smithi008.wqbvfi=up:active} 2024-04-14T13:34:59.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:59 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:34:59.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:59 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:34:59.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:59 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:34:59.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:34:59 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:34:59.939 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:34:59.939 INFO:teuthology.orchestra.run.smithi008.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-14T13:28:24.640841Z", "last_refresh": "2024-04-14T13:32:09.993071Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-14T13:28:19.222610Z", "last_refresh": "2024-04-14T13:32:09.993156Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-14T13:28:22.183609Z", "last_refresh": "2024-04-14T13:32:09.993226Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-14T13:34:52.574084Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-14T13:34:52.565217Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-14T13:28:17.214703Z", "last_refresh": "2024-04-14T13:32:09.992995Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi008:172.21.15.8=smithi008", "smithi012:172.21.15.12=smithi012"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-14T13:29:07.440988Z", "last_refresh": "2024-04-14T13:32:09.992849Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-14T13:28:23.319722Z", "last_refresh": "2024-04-14T13:32:09.993292Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-14T13:32:09.993421Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-14T13:28:20.882885Z", "last_refresh": "2024-04-14T13:32:09.993357Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-14T13:35:00.794 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:00 smithi012 ceph-mon[27236]: mds.? [v2:172.21.15.12:6832/1210020061,v1:172.21.15.12:6833/1210020061] up:boot 2024-04-14T13:35:00.794 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:00 smithi012 ceph-mon[27236]: fsmap foofs:1 {0=foofs.smithi008.wqbvfi=up:active} 1 up:standby 2024-04-14T13:35:00.794 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:00 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi012.vjovzp"}]: dispatch 2024-04-14T13:35:00.794 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:00 smithi012 ceph-mon[27236]: fsmap foofs:1 {0=foofs.smithi008.wqbvfi=up:active} 1 up:standby 2024-04-14T13:35:01.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:00 smithi008 ceph-mon[24786]: mds.? [v2:172.21.15.12:6832/1210020061,v1:172.21.15.12:6833/1210020061] up:boot 2024-04-14T13:35:01.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:00 smithi008 ceph-mon[24786]: fsmap foofs:1 {0=foofs.smithi008.wqbvfi=up:active} 1 up:standby 2024-04-14T13:35:01.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:00 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi012.vjovzp"}]: dispatch 2024-04-14T13:35:01.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:00 smithi008 ceph-mon[24786]: fsmap foofs:1 {0=foofs.smithi008.wqbvfi=up:active} 1 up:standby 2024-04-14T13:35:01.163 INFO:tasks.cephadm:mds.foofs has 0/2 2024-04-14T13:35:01.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:01 smithi012 ceph-mon[27236]: pgmap v216: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 418 B/s wr, 2 op/s 2024-04-14T13:35:01.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:01 smithi012 ceph-mon[27236]: from='client.14510 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:35:01.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:01 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:02.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:01 smithi008 ceph-mon[24786]: pgmap v216: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 418 B/s wr, 2 op/s 2024-04-14T13:35:02.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:01 smithi008 ceph-mon[24786]: from='client.14510 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:35:02.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:01 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:02.164 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph orch ls -f json 2024-04-14T13:35:03.201 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:02 smithi008 ceph-mon[24786]: pgmap v217: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-04-14T13:35:03.201 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:02 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:03.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:02 smithi012 ceph-mon[27236]: pgmap v217: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-04-14T13:35:03.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:02 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:05.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:04 smithi008 ceph-mon[24786]: pgmap v218: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-14T13:35:05.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:04 smithi012 ceph-mon[27236]: pgmap v218: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-14T13:35:05.843 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:35:05.844 INFO:teuthology.orchestra.run.smithi008.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-14T13:28:24.640841Z", "last_refresh": "2024-04-14T13:35:05.700662Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-14T13:28:19.222610Z", "last_refresh": "2024-04-14T13:35:01.500276Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-14T13:28:22.183609Z", "last_refresh": "2024-04-14T13:35:05.700795Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-14T13:34:52.574084Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-14T13:34:52.565217Z", "last_refresh": "2024-04-14T13:35:01.500883Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-14T13:28:17.214703Z", "last_refresh": "2024-04-14T13:35:01.500418Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi008:172.21.15.8=smithi008", "smithi012:172.21.15.12=smithi012"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-14T13:29:07.440988Z", "last_refresh": "2024-04-14T13:35:01.500493Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-14T13:28:23.319722Z", "last_refresh": "2024-04-14T13:35:01.500560Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-14T13:35:01.500626Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-14T13:28:20.882885Z", "last_refresh": "2024-04-14T13:35:05.700924Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-14T13:35:06.540 INFO:tasks.cephadm:mds.foofs has 2/2 2024-04-14T13:35:06.540 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-14T13:35:06.549 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-14T13:35:06.549 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- bash -c 'ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2' 2024-04-14T13:35:06.959 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:06.959 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:06.959 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:35:06.959 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:35:06.960 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:35:06.960 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:06 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:06.960 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:06 smithi008 ceph-mon[24786]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:35:07.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:07.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:07.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:35:07.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:35:07.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:35:07.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:06 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:07.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:06 smithi012 ceph-mon[27236]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:35:07.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:07 smithi008 ceph-mon[24786]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-14T13:35:07.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:07 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:08.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:07 smithi012 ceph-mon[27236]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-14T13:35:08.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:07 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:09.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:08 smithi012 ceph-mon[27236]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-14T13:35:09.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:08 smithi012 ceph-mon[27236]: from='client.14518 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:09.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:08 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-04-14T13:35:09.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:08 smithi008 ceph-mon[24786]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-14T13:35:09.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:08 smithi008 ceph-mon[24786]: from='client.14518 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:09.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:08 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-04-14T13:35:09.939 INFO:teuthology.orchestra.run.smithi008.stdout:NFS Cluster Created Successfully 2024-04-14T13:35:10.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:09 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-04-14T13:35:10.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:09 smithi008 ceph-mon[24786]: osdmap e65: 8 total, 8 up, 8 in 2024-04-14T13:35:10.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:09 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-04-14T13:35:10.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:09 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-04-14T13:35:10.243 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:09 smithi008 ceph-mon[24786]: osdmap e66: 8 total, 8 up, 8 in 2024-04-14T13:35:10.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:09 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-04-14T13:35:10.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:09 smithi012 ceph-mon[27236]: osdmap e65: 8 total, 8 up, 8 in 2024-04-14T13:35:10.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:09 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-04-14T13:35:10.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:09 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-04-14T13:35:10.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:09 smithi012 ceph-mon[27236]: osdmap e66: 8 total, 8 up, 8 in 2024-04-14T13:35:10.653 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- bash -c 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake' 2024-04-14T13:35:11.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: pgmap v222: 97 pgs: 15 creating+peering, 17 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 2 op/s 2024-04-14T13:35:11.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: Saving service nfs.foo spec with placement count:2 2024-04-14T13:35:11.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:11.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:35:11.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:35:11.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:35:11.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:11.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: Creating key for client.nfs.foo.0.0.smithi008.vefdqq 2024-04-14T13:35:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.vefdqq", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-14T13:35:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.vefdqq", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-14T13:35:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-14T13:35:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-14T13:35:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-14T13:35:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:35:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-14T13:35:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-14T13:35:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: Rados config object exists: conf-nfs.foo 2024-04-14T13:35:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: Creating key for client.nfs.foo.0.0.smithi008.vefdqq-rgw 2024-04-14T13:35:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.vefdqq-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T13:35:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.vefdqq-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T13:35:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:35:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: Deploying daemon nfs.foo.0.0.smithi008.vefdqq on smithi008 2024-04-14T13:35:11.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:10 smithi012 ceph-mon[27236]: osdmap e67: 8 total, 8 up, 8 in 2024-04-14T13:35:11.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: pgmap v222: 97 pgs: 15 creating+peering, 17 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 2 op/s 2024-04-14T13:35:11.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: Saving service nfs.foo spec with placement count:2 2024-04-14T13:35:11.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:11.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:35:11.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:35:11.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:35:11.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:11.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:11.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: Creating key for client.nfs.foo.0.0.smithi008.vefdqq 2024-04-14T13:35:11.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.vefdqq", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-14T13:35:11.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.vefdqq", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-14T13:35:11.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-14T13:35:11.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-14T13:35:11.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-14T13:35:11.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:35:11.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-14T13:35:11.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-14T13:35:11.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: Rados config object exists: conf-nfs.foo 2024-04-14T13:35:11.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: Creating key for client.nfs.foo.0.0.smithi008.vefdqq-rgw 2024-04-14T13:35:11.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.vefdqq-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T13:35:11.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.vefdqq-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T13:35:11.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:35:11.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: Deploying daemon nfs.foo.0.0.smithi008.vefdqq on smithi008 2024-04-14T13:35:11.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:10 smithi008 ceph-mon[24786]: osdmap e67: 8 total, 8 up, 8 in 2024-04-14T13:35:13.194 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:12 smithi008 ceph-mon[24786]: pgmap v225: 97 pgs: 15 creating+peering, 9 unknown, 73 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:35:13.194 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:12 smithi008 ceph-mon[24786]: mgrmap e20: smithi008.yemlhr(active, since 5m), standbys: smithi012.ksvnxm 2024-04-14T13:35:13.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:12 smithi012 ceph-mon[27236]: pgmap v225: 97 pgs: 15 creating+peering, 9 unknown, 73 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:35:13.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:12 smithi012 ceph-mon[27236]: mgrmap e20: smithi008.yemlhr(active, since 5m), standbys: smithi012.ksvnxm 2024-04-14T13:35:13.657 INFO:teuthology.orchestra.run.smithi008.stderr:Invalid command: Unexpected argument '--clusterid' 2024-04-14T13:35:13.658 INFO:teuthology.orchestra.run.smithi008.stderr:nfs export create cephfs [] [--readonly] [--client_addr ...] [--squash ] : Create a CephFS export 2024-04-14T13:35:13.658 INFO:teuthology.orchestra.run.smithi008.stderr:Error EINVAL: invalid command 2024-04-14T13:35:14.034 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:35:14.034 INFO:teuthology.orchestra.run.smithi008.stdout: "bind": "/fake", 2024-04-14T13:35:14.034 INFO:teuthology.orchestra.run.smithi008.stdout: "fs": "foofs", 2024-04-14T13:35:14.034 INFO:teuthology.orchestra.run.smithi008.stdout: "path": "/", 2024-04-14T13:35:14.034 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster": "foo", 2024-04-14T13:35:14.034 INFO:teuthology.orchestra.run.smithi008.stdout: "mode": "RW" 2024-04-14T13:35:14.034 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:35:14.638 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- bash -c 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done' 2024-04-14T13:35:15.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:15 smithi008 ceph-mon[24786]: pgmap v226: 97 pgs: 15 creating+peering, 82 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-14T13:35:15.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:15 smithi008 ceph-mon[24786]: from='client.14540 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:15.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:15 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:15.197 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:15 smithi008 ceph-mon[24786]: Creating key for client.nfs.foo.1.0.smithi012.meswmn 2024-04-14T13:35:15.197 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:15 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi012.meswmn", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-14T13:35:15.197 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:15 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi012.meswmn", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-14T13:35:15.197 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:15 smithi008 ceph-mon[24786]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-14T13:35:15.197 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:15 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-14T13:35:15.197 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:15 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-14T13:35:15.197 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:15 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-04-14T13:35:15.197 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:15 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:35:15.197 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:15 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-04-14T13:35:15.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:15 smithi012 ceph-mon[27236]: pgmap v226: 97 pgs: 15 creating+peering, 82 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-14T13:35:15.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:15 smithi012 ceph-mon[27236]: from='client.14540 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:15.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:15 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:15.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:15 smithi012 ceph-mon[27236]: Creating key for client.nfs.foo.1.0.smithi012.meswmn 2024-04-14T13:35:15.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:15 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi012.meswmn", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-14T13:35:15.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:15 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi012.meswmn", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-14T13:35:15.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:15 smithi012 ceph-mon[27236]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-14T13:35:15.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:15 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-14T13:35:15.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:15 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-14T13:35:15.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:15 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-04-14T13:35:15.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:15 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:35:15.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:15 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-04-14T13:35:17.713 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:17 smithi012 ceph-mon[27236]: pgmap v227: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 146 B/s rd, 292 B/s wr, 0 op/s 2024-04-14T13:35:17.713 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:17 smithi012 ceph-mon[27236]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:17.713 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:17 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-14T13:35:17.713 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:17 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-14T13:35:17.714 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:17 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi012.meswmn-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T13:35:17.714 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:17 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi012.meswmn-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T13:35:17.714 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:17 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:35:17.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:17 smithi008 ceph-mon[24786]: pgmap v227: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 146 B/s rd, 292 B/s wr, 0 op/s 2024-04-14T13:35:17.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:17 smithi008 ceph-mon[24786]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:17.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:17 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-14T13:35:17.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:17 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-14T13:35:17.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:17 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi012.meswmn-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T13:35:17.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:17 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi012.meswmn-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T13:35:17.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:17 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:35:18.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:18 smithi012 ceph-mon[27236]: Rados config object exists: conf-nfs.foo 2024-04-14T13:35:18.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:18 smithi012 ceph-mon[27236]: Creating key for client.nfs.foo.1.0.smithi012.meswmn-rgw 2024-04-14T13:35:18.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:18 smithi012 ceph-mon[27236]: Deploying daemon nfs.foo.1.0.smithi012.meswmn on smithi012 2024-04-14T13:35:18.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:18 smithi012 ceph-mon[27236]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:18.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:18 smithi008 ceph-mon[24786]: Rados config object exists: conf-nfs.foo 2024-04-14T13:35:18.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:18 smithi008 ceph-mon[24786]: Creating key for client.nfs.foo.1.0.smithi012.meswmn-rgw 2024-04-14T13:35:18.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:18 smithi008 ceph-mon[24786]: Deploying daemon nfs.foo.1.0.smithi012.meswmn on smithi012 2024-04-14T13:35:18.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:18 smithi008 ceph-mon[24786]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:19.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:19 smithi012 ceph-mon[27236]: pgmap v228: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 383 B/s wr, 0 op/s 2024-04-14T13:35:19.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:19 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:19.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:19 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:35:19.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:19 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:35:19.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:19 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:35:19.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:19 smithi008 ceph-mon[24786]: pgmap v228: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 383 B/s wr, 0 op/s 2024-04-14T13:35:19.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:19 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:19.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:19 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:35:19.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:19 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:35:19.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:19 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:35:20.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:20 smithi012 ceph-mon[27236]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:20.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:20 smithi008 ceph-mon[24786]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:21.806 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:21 smithi012 ceph-mon[27236]: pgmap v229: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-14T13:35:21.806 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:21 smithi012 ceph-mon[27236]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:21.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:21 smithi008 ceph-mon[24786]: pgmap v229: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-14T13:35:21.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:21 smithi008 ceph-mon[24786]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:22.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:22 smithi008 ceph-mon[24786]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:22.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:22 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:22.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:22 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:23.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:22 smithi012 ceph-mon[27236]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:23.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:22 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:23.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:22 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:23.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:23 smithi008 ceph-mon[24786]: pgmap v230: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-14T13:35:24.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:23 smithi012 ceph-mon[27236]: pgmap v230: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-14T13:35:24.368 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo ?:2049 2/2 2s ago 14s count:2 2024-04-14T13:35:24.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:24 smithi008 ceph-mon[24786]: from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:24.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:24 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:24.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:24 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:24.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:24 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:24.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:24 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:35:24.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:24 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:35:24.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:24 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:35:24.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:24 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:24.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:24 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:25.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:24 smithi012 ceph-mon[27236]: from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:25.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:24 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:25.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:24 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:25.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:24 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:25.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:24 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:35:25.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:24 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:35:25.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:24 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:35:25.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:24 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:25.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:24 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:25.094 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-14T13:35:25.103 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-14T13:35:25.104 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mkdir /mnt/foo' 2024-04-14T13:35:25.133 INFO:teuthology.orchestra.run.smithi008.stderr:+ mkdir /mnt/foo 2024-04-14T13:35:25.137 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done' 2024-04-14T13:35:25.204 INFO:teuthology.orchestra.run.smithi008.stderr:++ hostname 2024-04-14T13:35:25.204 INFO:teuthology.orchestra.run.smithi008.stderr:+ mount -t nfs smithi008:/fake /mnt/foo -o sync 2024-04-14T13:35:25.645 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'echo test > /mnt/foo/testfile' 2024-04-14T13:35:25.713 INFO:teuthology.orchestra.run.smithi008.stderr:+ echo test 2024-04-14T13:35:25.759 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c sync 2024-04-14T13:35:25.823 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:25 smithi008 ceph-mon[24786]: pgmap v231: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-04-14T13:35:25.823 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:25 smithi008 ceph-mon[24786]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:25.825 INFO:teuthology.orchestra.run.smithi008.stderr:+ sync 2024-04-14T13:35:26.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:25 smithi012 ceph-mon[27236]: pgmap v231: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-04-14T13:35:26.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:25 smithi012 ceph-mon[27236]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:26.100 INFO:teuthology.run_tasks:Running task parallel... 2024-04-14T13:35:26.129 INFO:teuthology.task.parallel:starting parallel... 2024-04-14T13:35:26.129 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-14T13:35:26.142 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-14T13:35:26.142 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-14T13:35:26.143 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-14T13:35:26.143 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-14T13:35:26.143 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-14T13:35:26.144 INFO:teuthology.task.exec:Executing custom commands... 2024-04-14T13:35:26.144 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-14T13:35:26.144 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 600 || true' 2024-04-14T13:35:26.187 INFO:teuthology.orchestra.run.smithi008.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-14T13:35:26.187 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:35:26.187 INFO:teuthology.orchestra.run.smithi008.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-14T13:35:26.189 INFO:teuthology.orchestra.run.smithi008.stdout:3 of 5 processes prepared for launch 0 sec 2024-04-14T13:35:26.289 INFO:teuthology.orchestra.run.smithi008.stdout:5 of 5 processes prepared for launch 0 sec 2024-04-14T13:35:26.289 INFO:teuthology.orchestra.run.smithi008.stdout:releasing clients 2024-04-14T13:35:27.289 INFO:teuthology.orchestra.run.smithi008.stdout: 5 296 54.56 MB/sec warmup 1 sec latency 39.559 ms 2024-04-14T13:35:27.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:27 smithi008 ceph-mon[24786]: pgmap v232: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-14T13:35:28.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:27 smithi012 ceph-mon[27236]: pgmap v232: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-14T13:35:28.289 INFO:teuthology.orchestra.run.smithi008.stdout: 5 605 51.57 MB/sec warmup 2 sec latency 13.235 ms 2024-04-14T13:35:29.292 INFO:teuthology.orchestra.run.smithi008.stdout: 5 1416 43.32 MB/sec warmup 3 sec latency 11.983 ms 2024-04-14T13:35:29.742 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:29 smithi008 ceph-mon[24786]: pgmap v233: 97 pgs: 97 active+clean; 9.2 MiB data, 52 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 786 KiB/s wr, 71 op/s 2024-04-14T13:35:30.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:29 smithi012 ceph-mon[27236]: pgmap v233: 97 pgs: 97 active+clean; 9.2 MiB data, 52 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 786 KiB/s wr, 71 op/s 2024-04-14T13:35:30.200 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-14T13:35:30.292 INFO:teuthology.orchestra.run.smithi008.stdout: 5 2507 35.92 MB/sec warmup 4 sec latency 9.049 ms 2024-04-14T13:35:31.292 INFO:teuthology.orchestra.run.smithi008.stdout: 5 3664 38.84 MB/sec warmup 5 sec latency 20.494 ms 2024-04-14T13:35:31.824 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:31 smithi008 ceph-mon[24786]: pgmap v234: 97 pgs: 97 active+clean; 60 MiB data, 115 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 5.2 MiB/s wr, 193 op/s 2024-04-14T13:35:32.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:31 smithi012 ceph-mon[27236]: pgmap v234: 97 pgs: 97 active+clean; 60 MiB data, 115 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 5.2 MiB/s wr, 193 op/s 2024-04-14T13:35:32.292 INFO:teuthology.orchestra.run.smithi008.stdout: 5 4033 33.39 MB/sec warmup 6 sec latency 170.226 ms 2024-04-14T13:35:32.583 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]: dispatch 2024-04-14T13:35:32.583 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-04-14T13:35:32.583 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-14T13:35:32.583 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-04-14T13:35:32.583 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:35:32.583 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:35:33.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]: dispatch 2024-04-14T13:35:33.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-04-14T13:35:33.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-14T13:35:33.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-04-14T13:35:33.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:35:33.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:35:33.292 INFO:teuthology.orchestra.run.smithi008.stdout: 5 4663 29.96 MB/sec warmup 7 sec latency 21.351 ms 2024-04-14T13:35:33.591 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:33 smithi008 ceph-mon[24786]: pgmap v235: 97 pgs: 97 active+clean; 115 MiB data, 173 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 11 MiB/s wr, 535 op/s 2024-04-14T13:35:33.591 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:33 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]': finished 2024-04-14T13:35:33.591 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:33 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-04-14T13:35:33.591 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:33 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-04-14T13:35:33.591 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:33 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-04-14T13:35:33.591 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:33 smithi008 ceph-mon[24786]: osdmap e68: 8 total, 8 up, 8 in 2024-04-14T13:35:34.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:33 smithi012 ceph-mon[27236]: pgmap v235: 97 pgs: 97 active+clean; 115 MiB data, 173 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 11 MiB/s wr, 535 op/s 2024-04-14T13:35:34.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:33 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]': finished 2024-04-14T13:35:34.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:33 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-04-14T13:35:34.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:33 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-04-14T13:35:34.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:33 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-04-14T13:35:34.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:33 smithi012 ceph-mon[27236]: osdmap e68: 8 total, 8 up, 8 in 2024-04-14T13:35:34.292 INFO:teuthology.orchestra.run.smithi008.stdout: 5 5510 28.48 MB/sec warmup 8 sec latency 20.335 ms 2024-04-14T13:35:34.599 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:34 smithi008 ceph-mon[24786]: osdmap e69: 8 total, 8 up, 8 in 2024-04-14T13:35:35.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:34 smithi012 ceph-mon[27236]: osdmap e69: 8 total, 8 up, 8 in 2024-04-14T13:35:35.292 INFO:teuthology.orchestra.run.smithi008.stdout: 5 6170 27.13 MB/sec warmup 9 sec latency 18.595 ms 2024-04-14T13:35:35.522 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-14T13:35:35.610 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:35 smithi008 ceph-mon[24786]: pgmap v238: 97 pgs: 97 active+clean; 126 MiB data, 198 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 19 MiB/s wr, 1.01k op/s 2024-04-14T13:35:36.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:35 smithi012 ceph-mon[27236]: pgmap v238: 97 pgs: 97 active+clean; 126 MiB data, 198 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 19 MiB/s wr, 1.01k op/s 2024-04-14T13:35:36.292 INFO:teuthology.orchestra.run.smithi008.stdout: 5 6910 27.47 MB/sec warmup 10 sec latency 58.139 ms 2024-04-14T13:35:37.293 INFO:teuthology.orchestra.run.smithi008.stdout: 5 7458 26.84 MB/sec warmup 11 sec latency 32.788 ms 2024-04-14T13:35:37.674 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:37 smithi008 ceph-mon[24786]: pgmap v239: 97 pgs: 97 active+clean; 127 MiB data, 225 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 19 MiB/s wr, 969 op/s; 0 B/s, 0 objects/s recovering 2024-04-14T13:35:38.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:37 smithi012 ceph-mon[27236]: pgmap v239: 97 pgs: 97 active+clean; 127 MiB data, 225 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 19 MiB/s wr, 969 op/s; 0 B/s, 0 objects/s recovering 2024-04-14T13:35:38.293 INFO:teuthology.orchestra.run.smithi008.stdout: 5 8047 25.06 MB/sec warmup 12 sec latency 24.173 ms 2024-04-14T13:35:39.293 INFO:teuthology.orchestra.run.smithi008.stdout: 5 8655 24.74 MB/sec warmup 13 sec latency 43.888 ms 2024-04-14T13:35:39.691 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:39 smithi008 ceph-mon[24786]: pgmap v240: 97 pgs: 97 active+clean; 133 MiB data, 302 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 15 MiB/s wr, 1.07k op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-14T13:35:40.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:39 smithi012 ceph-mon[27236]: pgmap v240: 97 pgs: 97 active+clean; 133 MiB data, 302 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 15 MiB/s wr, 1.07k op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-14T13:35:40.293 INFO:teuthology.orchestra.run.smithi008.stdout: 5 9372 23.35 MB/sec warmup 14 sec latency 27.171 ms 2024-04-14T13:35:40.723 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph mgr module enable nfs --force' 2024-04-14T13:35:41.293 INFO:teuthology.orchestra.run.smithi008.stdout: 5 10007 23.64 MB/sec warmup 15 sec latency 39.569 ms 2024-04-14T13:35:41.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:41 smithi008 ceph-mon[24786]: pgmap v241: 97 pgs: 97 active+clean; 139 MiB data, 332 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 9.2 MiB/s wr, 818 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-14T13:35:42.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:41 smithi012 ceph-mon[27236]: pgmap v241: 97 pgs: 97 active+clean; 139 MiB data, 332 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 9.2 MiB/s wr, 818 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-14T13:35:42.293 INFO:teuthology.orchestra.run.smithi008.stdout: 5 10575 23.41 MB/sec warmup 16 sec latency 57.968 ms 2024-04-14T13:35:43.293 INFO:teuthology.orchestra.run.smithi008.stdout: 5 11059 23.03 MB/sec warmup 17 sec latency 22.899 ms 2024-04-14T13:35:43.816 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:43 smithi008 ceph-mon[24786]: pgmap v242: 97 pgs: 97 active+clean; 147 MiB data, 385 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 11 MiB/s wr, 1.04k op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-14T13:35:44.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:43 smithi012 ceph-mon[27236]: pgmap v242: 97 pgs: 97 active+clean; 147 MiB data, 385 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 11 MiB/s wr, 1.04k op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-14T13:35:44.293 INFO:teuthology.orchestra.run.smithi008.stdout: 5 11622 22.04 MB/sec warmup 18 sec latency 25.230 ms 2024-04-14T13:35:44.627 INFO:teuthology.orchestra.run.smithi008.stderr:module 'nfs' is already enabled 2024-04-14T13:35:44.830 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:44 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2858559963' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-14T13:35:44.830 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:44 smithi008 ceph-mon[24786]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-14T13:35:45.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:44 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/2858559963' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-14T13:35:45.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:44 smithi012 ceph-mon[27236]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-14T13:35:45.293 INFO:teuthology.orchestra.run.smithi008.stdout: 5 12317 21.99 MB/sec warmup 19 sec latency 20.724 ms 2024-04-14T13:35:45.660 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-14T13:35:45.834 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:45 smithi008 ceph-mon[24786]: pgmap v243: 97 pgs: 97 active+clean; 147 MiB data, 396 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 858 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-14T13:35:45.834 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:45 smithi008 ceph-mon[24786]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-04-14T13:35:45.834 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:45 smithi008 ceph-mon[24786]: mgrmap e21: smithi008.yemlhr(active, since 6m), standbys: smithi012.ksvnxm 2024-04-14T13:35:45.950 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:45 smithi012 ceph-mon[27236]: pgmap v243: 97 pgs: 97 active+clean; 147 MiB data, 396 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 858 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-14T13:35:45.950 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:45 smithi012 ceph-mon[27236]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-04-14T13:35:45.950 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:45 smithi012 ceph-mon[27236]: mgrmap e21: smithi008.yemlhr(active, since 6m), standbys: smithi012.ksvnxm 2024-04-14T13:35:46.293 INFO:teuthology.orchestra.run.smithi008.stdout: 5 13047 21.36 MB/sec warmup 20 sec latency 27.203 ms 2024-04-14T13:35:47.293 INFO:teuthology.orchestra.run.smithi008.stdout: 5 13547 21.43 MB/sec warmup 21 sec latency 54.841 ms 2024-04-14T13:35:47.649 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:47 smithi008 ceph-mon[24786]: pgmap v244: 97 pgs: 97 active+clean; 149 MiB data, 402 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 8.4 MiB/s wr, 746 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-14T13:35:47.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:47 smithi012 ceph-mon[27236]: pgmap v244: 97 pgs: 97 active+clean; 149 MiB data, 402 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 8.4 MiB/s wr, 746 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-14T13:35:48.293 INFO:teuthology.orchestra.run.smithi008.stdout: 5 14046 21.23 MB/sec warmup 22 sec latency 33.345 ms 2024-04-14T13:35:49.090 INFO:teuthology.orchestra.run.smithi008.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T13:35:49.293 INFO:teuthology.orchestra.run.smithi008.stdout: 5 14480 21.12 MB/sec warmup 23 sec latency 34.733 ms 2024-04-14T13:35:49.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:49 smithi008 ceph-mon[24786]: pgmap v245: 97 pgs: 97 active+clean; 154 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9.5 MiB/s wr, 886 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-14T13:35:49.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:49 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:49.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:49 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:35:49.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:49 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:35:49.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:49 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:35:49.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:49 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:49.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:49 smithi012 ceph-mon[27236]: pgmap v245: 97 pgs: 97 active+clean; 154 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9.5 MiB/s wr, 886 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-14T13:35:49.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:49 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:49.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:49 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:35:49.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:49 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:35:49.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:49 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:35:49.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:49 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:50.294 INFO:teuthology.orchestra.run.smithi008.stdout: 5 14932 20.36 MB/sec warmup 24 sec latency 26.260 ms 2024-04-14T13:35:50.298 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-14T13:35:50.299 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-14T13:35:50.299 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-14T13:35:50.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:50 smithi008 ceph-mon[24786]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:50.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:50 smithi008 ceph-mon[24786]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T13:35:50.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:50 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:50.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:50 smithi008 ceph-mon[24786]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T13:35:51.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:50 smithi012 ceph-mon[27236]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:51.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:50 smithi012 ceph-mon[27236]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T13:35:51.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:50 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:35:51.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:50 smithi012 ceph-mon[27236]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T13:35:51.294 INFO:teuthology.orchestra.run.smithi008.stdout: 5 15658 20.39 MB/sec warmup 25 sec latency 21.675 ms 2024-04-14T13:35:51.677 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:51 smithi008 ceph-mon[24786]: pgmap v246: 97 pgs: 97 active+clean; 156 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 9.1 MiB/s wr, 864 op/s 2024-04-14T13:35:52.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:51 smithi012 ceph-mon[27236]: pgmap v246: 97 pgs: 97 active+clean; 156 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 9.1 MiB/s wr, 864 op/s 2024-04-14T13:35:52.294 INFO:teuthology.orchestra.run.smithi008.stdout: 5 16517 20.00 MB/sec warmup 26 sec latency 24.338 ms 2024-04-14T13:35:53.295 INFO:teuthology.orchestra.run.smithi008.stdout: 5 17004 20.06 MB/sec warmup 27 sec latency 33.025 ms 2024-04-14T13:35:53.699 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:53 smithi008 ceph-mon[24786]: pgmap v247: 97 pgs: 97 active+clean; 167 MiB data, 530 MiB used, 715 GiB / 715 GiB avail; 8.4 KiB/s rd, 9.2 MiB/s wr, 875 op/s 2024-04-14T13:35:54.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:53 smithi012 ceph-mon[27236]: pgmap v247: 97 pgs: 97 active+clean; 167 MiB data, 530 MiB used, 715 GiB / 715 GiB avail; 8.4 KiB/s rd, 9.2 MiB/s wr, 875 op/s 2024-04-14T13:35:54.295 INFO:teuthology.orchestra.run.smithi008.stdout: 5 17590 20.16 MB/sec warmup 28 sec latency 59.001 ms 2024-04-14T13:35:54.999 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:35:55.295 INFO:teuthology.orchestra.run.smithi008.stdout: 5 18028 20.06 MB/sec warmup 29 sec latency 26.735 ms 2024-04-14T13:35:55.713 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:55 smithi008 ceph-mon[24786]: pgmap v248: 97 pgs: 97 active+clean; 171 MiB data, 546 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 8.1 MiB/s wr, 749 op/s 2024-04-14T13:35:56.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:55 smithi012 ceph-mon[27236]: pgmap v248: 97 pgs: 97 active+clean; 171 MiB data, 546 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 8.1 MiB/s wr, 749 op/s 2024-04-14T13:35:56.091 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:35:56.092 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (5m) 31s ago 6m 23.7M - ba2b418f427c fbc215a6ff5d 2024-04-14T13:35:56.092 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (6m) 31s ago 6m 6405k - 17.2.0 e1d6a67b021e 68b810a9ff93 2024-04-14T13:35:56.092 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (6m) 33s ago 6m 6400k - 17.2.0 e1d6a67b021e 6bc31c51b938 2024-04-14T13:35:56.092 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (5m) 31s ago 6m 54.0M - 8.3.5 dad864ee21e9 c7b42d05cedc 2024-04-14T13:35:56.092 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (60s) 31s ago 59s 14.9M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:35:56.092 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (57s) 33s ago 56s 11.9M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:35:56.092 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:9283 running (8m) 31s ago 8m 460M - 17.2.0 e1d6a67b021e 074f98e540c3 2024-04-14T13:35:56.092 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283 running (6m) 33s ago 6m 412M - 17.2.0 e1d6a67b021e 596e23b54c3d 2024-04-14T13:35:56.092 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (8m) 31s ago 8m 49.7M 2048M 17.2.0 e1d6a67b021e 7786c27a60d0 2024-04-14T13:35:56.092 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (6m) 33s ago 6m 34.5M 2048M 17.2.0 e1d6a67b021e f6cd483dfa31 2024-04-14T13:35:56.092 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (42s) 31s ago 42s 46.9M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:35:56.092 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (37s) 33s ago 36s 46.6M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:35:56.092 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (6m) 31s ago 6m 16.8M - 1dbe0e931976 2338da898186 2024-04-14T13:35:56.092 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (5m) 33s ago 5m 18.7M - 1dbe0e931976 c2fae00595dd 2024-04-14T13:35:56.093 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (5m) 31s ago 5m 43.7M 3217M 17.2.0 e1d6a67b021e 3b8cdea7890c 2024-04-14T13:35:56.093 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (4m) 31s ago 4m 43.8M 3217M 17.2.0 e1d6a67b021e be86642c7790 2024-04-14T13:35:56.093 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (4m) 31s ago 4m 39.6M 3217M 17.2.0 e1d6a67b021e b1017f00578f 2024-04-14T13:35:56.093 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (4m) 31s ago 4m 42.7M 3217M 17.2.0 e1d6a67b021e 66fca4b191be 2024-04-14T13:35:56.093 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (3m) 33s ago 3m 41.4M 3985M 17.2.0 e1d6a67b021e 43b02cb64951 2024-04-14T13:35:56.093 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (3m) 33s ago 3m 38.9M 3985M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:35:56.093 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (2m) 33s ago 2m 39.7M 3985M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:35:56.093 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (2m) 33s ago 2m 42.1M 3985M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:35:56.093 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (5m) 31s ago 5m 62.5M - 514e6a882f6e 11fca64796ed 2024-04-14T13:35:56.295 INFO:teuthology.orchestra.run.smithi008.stdout: 5 18473 19.47 MB/sec warmup 30 sec latency 21.711 ms 2024-04-14T13:35:56.719 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:56 smithi008 ceph-mon[24786]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:56.719 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:56 smithi008 ceph-mon[24786]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:56.743 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:35:56.744 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:35:56.744 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:35:56.744 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:35:56.744 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:35:56.744 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:35:56.744 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:35:56.744 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:35:56.744 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T13:35:56.744 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:35:56.745 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:35:56.745 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:35:56.745 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:35:56.745 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:35:56.745 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-14T13:35:56.745 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:35:56.745 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:35:57.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:56 smithi012 ceph-mon[27236]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:57.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:56 smithi012 ceph-mon[27236]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:57.295 INFO:teuthology.orchestra.run.smithi008.stdout: 5 19081 19.33 MB/sec warmup 31 sec latency 18.059 ms 2024-04-14T13:35:57.299 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:35:57.299 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T13:35:57.299 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:35:57.299 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-04-14T13:35:57.299 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "", 2024-04-14T13:35:57.299 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 image" 2024-04-14T13:35:57.299 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:35:57.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:57 smithi008 ceph-mon[24786]: pgmap v249: 97 pgs: 97 active+clean; 168 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 7.7 MiB/s wr, 727 op/s 2024-04-14T13:35:57.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:57 smithi008 ceph-mon[24786]: from='client.24413 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:57.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:57 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/77680764' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:35:57.958 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-04-14T13:35:58.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:57 smithi012 ceph-mon[27236]: pgmap v249: 97 pgs: 97 active+clean; 168 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 7.7 MiB/s wr, 727 op/s 2024-04-14T13:35:58.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:57 smithi012 ceph-mon[27236]: from='client.24413 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:58.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:57 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/77680764' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:35:58.295 INFO:teuthology.orchestra.run.smithi008.stdout: 5 19825 19.10 MB/sec warmup 32 sec latency 28.932 ms 2024-04-14T13:35:58.728 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:58 smithi008 ceph-mon[24786]: from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:58.729 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:58 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/1788218567' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:35:59.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:58 smithi012 ceph-mon[27236]: from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:35:59.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:58 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/1788218567' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:35:59.295 INFO:teuthology.orchestra.run.smithi008.stdout: 5 20481 19.12 MB/sec warmup 33 sec latency 22.738 ms 2024-04-14T13:35:59.739 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:35:59 smithi008 ceph-mon[24786]: pgmap v250: 97 pgs: 97 active+clean; 175 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 9.0 MiB/s wr, 850 op/s 2024-04-14T13:36:00.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:35:59 smithi012 ceph-mon[27236]: pgmap v250: 97 pgs: 97 active+clean; 175 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 9.0 MiB/s wr, 850 op/s 2024-04-14T13:36:00.295 INFO:teuthology.orchestra.run.smithi008.stdout: 5 21341 19.64 MB/sec warmup 34 sec latency 14.642 ms 2024-04-14T13:36:01.295 INFO:teuthology.orchestra.run.smithi008.stdout: 5 21813 19.39 MB/sec warmup 35 sec latency 21.471 ms 2024-04-14T13:36:01.816 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:01 smithi008 ceph-mon[24786]: pgmap v251: 97 pgs: 97 active+clean; 181 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 8.8 MiB/s wr, 907 op/s 2024-04-14T13:36:02.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:01 smithi012 ceph-mon[27236]: pgmap v251: 97 pgs: 97 active+clean; 181 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 8.8 MiB/s wr, 907 op/s 2024-04-14T13:36:02.295 INFO:teuthology.orchestra.run.smithi008.stdout: 5 22403 19.07 MB/sec warmup 36 sec latency 18.967 ms 2024-04-14T13:36:03.295 INFO:teuthology.orchestra.run.smithi008.stdout: 5 23122 19.07 MB/sec warmup 37 sec latency 18.036 ms 2024-04-14T13:36:03.843 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:03 smithi008 ceph-mon[24786]: pgmap v252: 97 pgs: 97 active+clean; 180 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 8.7 MiB/s wr, 798 op/s 2024-04-14T13:36:04.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:03 smithi012 ceph-mon[27236]: pgmap v252: 97 pgs: 97 active+clean; 180 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 8.7 MiB/s wr, 798 op/s 2024-04-14T13:36:04.295 INFO:teuthology.orchestra.run.smithi008.stdout: 5 23711 18.79 MB/sec warmup 38 sec latency 35.979 ms 2024-04-14T13:36:05.295 INFO:teuthology.orchestra.run.smithi008.stdout: 5 24268 18.94 MB/sec warmup 39 sec latency 25.441 ms 2024-04-14T13:36:05.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:05 smithi008 ceph-mon[24786]: pgmap v253: 97 pgs: 97 active+clean; 184 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 7.9 MiB/s wr, 738 op/s 2024-04-14T13:36:06.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:05 smithi012 ceph-mon[27236]: pgmap v253: 97 pgs: 97 active+clean; 184 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 7.9 MiB/s wr, 738 op/s 2024-04-14T13:36:06.302 INFO:teuthology.orchestra.run.smithi008.stdout: 5 24900 19.10 MB/sec warmup 40 sec latency 39.911 ms 2024-04-14T13:36:07.302 INFO:teuthology.orchestra.run.smithi008.stdout: 5 25253 18.87 MB/sec warmup 41 sec latency 30.207 ms 2024-04-14T13:36:07.889 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:07 smithi008 ceph-mon[24786]: pgmap v254: 97 pgs: 97 active+clean; 187 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 7.8 MiB/s wr, 697 op/s 2024-04-14T13:36:08.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:07 smithi012 ceph-mon[27236]: pgmap v254: 97 pgs: 97 active+clean; 187 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 7.8 MiB/s wr, 697 op/s 2024-04-14T13:36:08.302 INFO:teuthology.orchestra.run.smithi008.stdout: 5 25768 18.51 MB/sec warmup 42 sec latency 33.656 ms 2024-04-14T13:36:09.302 INFO:teuthology.orchestra.run.smithi008.stdout: 5 26422 18.60 MB/sec warmup 43 sec latency 27.265 ms 2024-04-14T13:36:09.916 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:09 smithi008 ceph-mon[24786]: pgmap v255: 97 pgs: 97 active+clean; 195 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.6 MiB/s wr, 920 op/s 2024-04-14T13:36:10.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:09 smithi012 ceph-mon[27236]: pgmap v255: 97 pgs: 97 active+clean; 195 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.6 MiB/s wr, 920 op/s 2024-04-14T13:36:10.302 INFO:teuthology.orchestra.run.smithi008.stdout: 5 27159 18.31 MB/sec warmup 44 sec latency 35.974 ms 2024-04-14T13:36:11.302 INFO:teuthology.orchestra.run.smithi008.stdout: 5 27993 18.67 MB/sec warmup 45 sec latency 33.460 ms 2024-04-14T13:36:11.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:11 smithi008 ceph-mon[24786]: pgmap v256: 97 pgs: 97 active+clean; 205 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.5 MiB/s wr, 957 op/s 2024-04-14T13:36:12.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:11 smithi012 ceph-mon[27236]: pgmap v256: 97 pgs: 97 active+clean; 205 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.5 MiB/s wr, 957 op/s 2024-04-14T13:36:12.302 INFO:teuthology.orchestra.run.smithi008.stdout: 5 28688 18.87 MB/sec warmup 46 sec latency 22.656 ms 2024-04-14T13:36:13.302 INFO:teuthology.orchestra.run.smithi008.stdout: 5 29357 18.59 MB/sec warmup 47 sec latency 28.810 ms 2024-04-14T13:36:13.939 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:13 smithi008 ceph-mon[24786]: pgmap v257: 97 pgs: 97 active+clean; 207 MiB data, 838 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 9.5 MiB/s wr, 867 op/s 2024-04-14T13:36:14.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:13 smithi012 ceph-mon[27236]: pgmap v257: 97 pgs: 97 active+clean; 207 MiB data, 838 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 9.5 MiB/s wr, 867 op/s 2024-04-14T13:36:14.302 INFO:teuthology.orchestra.run.smithi008.stdout: 5 29988 18.66 MB/sec warmup 48 sec latency 25.522 ms 2024-04-14T13:36:15.302 INFO:teuthology.orchestra.run.smithi008.stdout: 5 30723 18.43 MB/sec warmup 49 sec latency 41.810 ms 2024-04-14T13:36:15.740 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:15 smithi008 ceph-mon[24786]: pgmap v258: 97 pgs: 97 active+clean; 208 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 856 op/s 2024-04-14T13:36:16.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:15 smithi012 ceph-mon[27236]: pgmap v258: 97 pgs: 97 active+clean; 208 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 856 op/s 2024-04-14T13:36:16.302 INFO:teuthology.orchestra.run.smithi008.stdout: 5 31354 18.63 MB/sec warmup 50 sec latency 19.512 ms 2024-04-14T13:36:17.302 INFO:teuthology.orchestra.run.smithi008.stdout: 5 32067 18.79 MB/sec warmup 51 sec latency 32.396 ms 2024-04-14T13:36:17.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:17 smithi008 ceph-mon[24786]: pgmap v259: 97 pgs: 97 active+clean; 212 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.6 MiB/s wr, 799 op/s 2024-04-14T13:36:18.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:17 smithi012 ceph-mon[27236]: pgmap v259: 97 pgs: 97 active+clean; 212 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.6 MiB/s wr, 799 op/s 2024-04-14T13:36:18.302 INFO:teuthology.orchestra.run.smithi008.stdout: 5 32382 18.56 MB/sec warmup 52 sec latency 36.650 ms 2024-04-14T13:36:19.302 INFO:teuthology.orchestra.run.smithi008.stdout: 5 32891 18.29 MB/sec warmup 53 sec latency 36.120 ms 2024-04-14T13:36:19.814 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:19 smithi008 ceph-mon[24786]: pgmap v260: 97 pgs: 97 active+clean; 221 MiB data, 926 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.8 MiB/s wr, 938 op/s 2024-04-14T13:36:20.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:19 smithi012 ceph-mon[27236]: pgmap v260: 97 pgs: 97 active+clean; 221 MiB data, 926 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.8 MiB/s wr, 938 op/s 2024-04-14T13:36:20.303 INFO:teuthology.orchestra.run.smithi008.stdout: 5 33477 18.35 MB/sec warmup 54 sec latency 22.221 ms 2024-04-14T13:36:21.303 INFO:teuthology.orchestra.run.smithi008.stdout: 5 34173 18.09 MB/sec warmup 55 sec latency 68.034 ms 2024-04-14T13:36:21.823 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:21 smithi008 ceph-mon[24786]: pgmap v261: 97 pgs: 97 active+clean; 226 MiB data, 957 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 9.6 MiB/s wr, 887 op/s 2024-04-14T13:36:22.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:21 smithi012 ceph-mon[27236]: pgmap v261: 97 pgs: 97 active+clean; 226 MiB data, 957 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 9.6 MiB/s wr, 887 op/s 2024-04-14T13:36:22.303 INFO:teuthology.orchestra.run.smithi008.stdout: 5 34709 18.12 MB/sec warmup 56 sec latency 26.925 ms 2024-04-14T13:36:23.303 INFO:teuthology.orchestra.run.smithi008.stdout: 5 35541 18.47 MB/sec warmup 57 sec latency 21.638 ms 2024-04-14T13:36:23.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:23 smithi008 ceph-mon[24786]: pgmap v262: 97 pgs: 97 active+clean; 225 MiB data, 998 MiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 9.3 MiB/s wr, 829 op/s 2024-04-14T13:36:24.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:23 smithi012 ceph-mon[27236]: pgmap v262: 97 pgs: 97 active+clean; 225 MiB data, 998 MiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 9.3 MiB/s wr, 829 op/s 2024-04-14T13:36:24.303 INFO:teuthology.orchestra.run.smithi008.stdout: 5 35905 18.29 MB/sec warmup 58 sec latency 24.025 ms 2024-04-14T13:36:25.303 INFO:teuthology.orchestra.run.smithi008.stdout: 5 36349 18.03 MB/sec warmup 59 sec latency 39.040 ms 2024-04-14T13:36:25.881 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:25 smithi008 ceph-mon[24786]: pgmap v263: 97 pgs: 97 active+clean; 226 MiB data, 1016 MiB used, 714 GiB / 715 GiB avail; 9.9 KiB/s rd, 8.4 MiB/s wr, 763 op/s 2024-04-14T13:36:26.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:25 smithi012 ceph-mon[27236]: pgmap v263: 97 pgs: 97 active+clean; 226 MiB data, 1016 MiB used, 714 GiB / 715 GiB avail; 9.9 KiB/s rd, 8.4 MiB/s wr, 763 op/s 2024-04-14T13:36:26.303 INFO:teuthology.orchestra.run.smithi008.stdout: 5 36978 18.10 MB/sec warmup 60 sec latency 39.135 ms 2024-04-14T13:36:27.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 37829 17.93 MB/sec warmup 61 sec latency 25.341 ms 2024-04-14T13:36:27.893 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:27 smithi008 ceph-mon[24786]: pgmap v264: 97 pgs: 97 active+clean; 226 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 7.9 MiB/s wr, 739 op/s 2024-04-14T13:36:28.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:27 smithi012 ceph-mon[27236]: pgmap v264: 97 pgs: 97 active+clean; 226 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 7.9 MiB/s wr, 739 op/s 2024-04-14T13:36:28.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 38482 18.10 MB/sec warmup 62 sec latency 48.511 ms 2024-04-14T13:36:28.485 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:36:29.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 39060 18.15 MB/sec warmup 63 sec latency 100.915 ms 2024-04-14T13:36:29.450 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:36:29.450 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (5m) 65s ago 7m 23.7M - ba2b418f427c fbc215a6ff5d 2024-04-14T13:36:29.450 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (7m) 65s ago 7m 6405k - 17.2.0 e1d6a67b021e 68b810a9ff93 2024-04-14T13:36:29.450 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (6m) 67s ago 6m 6400k - 17.2.0 e1d6a67b021e 6bc31c51b938 2024-04-14T13:36:29.450 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (5m) 65s ago 7m 54.0M - 8.3.5 dad864ee21e9 c7b42d05cedc 2024-04-14T13:36:29.450 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (93s) 65s ago 92s 14.9M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:36:29.450 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (90s) 67s ago 90s 11.9M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:36:29.450 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:9283 running (8m) 65s ago 8m 460M - 17.2.0 e1d6a67b021e 074f98e540c3 2024-04-14T13:36:29.450 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283 running (6m) 67s ago 6m 412M - 17.2.0 e1d6a67b021e 596e23b54c3d 2024-04-14T13:36:29.450 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (8m) 65s ago 8m 49.7M 2048M 17.2.0 e1d6a67b021e 7786c27a60d0 2024-04-14T13:36:29.450 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (6m) 67s ago 6m 34.5M 2048M 17.2.0 e1d6a67b021e f6cd483dfa31 2024-04-14T13:36:29.450 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (75s) 65s ago 75s 46.9M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:36:29.450 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (70s) 67s ago 70s 46.6M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:36:29.451 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (7m) 65s ago 7m 16.8M - 1dbe0e931976 2338da898186 2024-04-14T13:36:29.451 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (6m) 67s ago 6m 18.7M - 1dbe0e931976 c2fae00595dd 2024-04-14T13:36:29.451 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (5m) 65s ago 5m 43.7M 3217M 17.2.0 e1d6a67b021e 3b8cdea7890c 2024-04-14T13:36:29.451 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (5m) 65s ago 5m 43.8M 3217M 17.2.0 e1d6a67b021e be86642c7790 2024-04-14T13:36:29.451 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (5m) 65s ago 5m 39.6M 3217M 17.2.0 e1d6a67b021e b1017f00578f 2024-04-14T13:36:29.451 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (4m) 65s ago 4m 42.7M 3217M 17.2.0 e1d6a67b021e 66fca4b191be 2024-04-14T13:36:29.451 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (4m) 67s ago 4m 41.4M 3985M 17.2.0 e1d6a67b021e 43b02cb64951 2024-04-14T13:36:29.451 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (3m) 67s ago 3m 38.9M 3985M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:36:29.451 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (3m) 67s ago 3m 39.7M 3985M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:36:29.451 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (2m) 67s ago 2m 42.1M 3985M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:36:29.451 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (6m) 65s ago 6m 62.5M - 514e6a882f6e 11fca64796ed 2024-04-14T13:36:29.973 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:36:29.973 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:36:29.974 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:36:29.974 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:36:29.974 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:36:29.974 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:36:29.974 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:36:29.974 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:36:29.974 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T13:36:29.974 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:36:29.974 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:36:29.974 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:36:29.974 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:36:29.974 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:36:29.974 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-14T13:36:29.974 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:36:29.974 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:36:29.978 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:29 smithi008 ceph-mon[24786]: pgmap v265: 97 pgs: 97 active+clean; 240 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.3 KiB/s rd, 9.3 MiB/s wr, 903 op/s 2024-04-14T13:36:29.978 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:29 smithi008 ceph-mon[24786]: from='client.24429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:36:30.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:29 smithi012 ceph-mon[27236]: pgmap v265: 97 pgs: 97 active+clean; 240 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.3 KiB/s rd, 9.3 MiB/s wr, 903 op/s 2024-04-14T13:36:30.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:29 smithi012 ceph-mon[27236]: from='client.24429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:36:30.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 39398 18.06 MB/sec warmup 64 sec latency 151.019 ms 2024-04-14T13:36:30.503 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:36:30.503 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T13:36:30.504 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:36:30.504 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-04-14T13:36:30.504 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "", 2024-04-14T13:36:30.504 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 image" 2024-04-14T13:36:30.504 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:36:30.983 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:30 smithi008 ceph-mon[24786]: from='client.14650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:36:30.983 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:30 smithi008 ceph-mon[24786]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:36:30.983 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:30 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2953323218' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:36:31.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:30 smithi012 ceph-mon[27236]: from='client.14650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:36:31.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:30 smithi012 ceph-mon[27236]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:36:31.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:30 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/2953323218' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:36:31.148 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-04-14T13:36:31.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 39884 17.83 MB/sec warmup 65 sec latency 43.192 ms 2024-04-14T13:36:31.988 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:31 smithi008 ceph-mon[24786]: pgmap v266: 97 pgs: 97 active+clean; 244 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 9.0 MiB/s wr, 907 op/s 2024-04-14T13:36:31.988 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:31 smithi008 ceph-mon[24786]: from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:36:31.988 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:31 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2347859953' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:36:32.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:31 smithi012 ceph-mon[27236]: pgmap v266: 97 pgs: 97 active+clean; 244 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 9.0 MiB/s wr, 907 op/s 2024-04-14T13:36:32.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:31 smithi012 ceph-mon[27236]: from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:36:32.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:31 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/2347859953' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:36:32.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 40509 17.82 MB/sec warmup 66 sec latency 29.513 ms 2024-04-14T13:36:32.943 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:36:32.943 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:36:33.024 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:36:33.024 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:36:33.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 41342 17.74 MB/sec warmup 67 sec latency 50.788 ms 2024-04-14T13:36:33.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:33 smithi008 ceph-mon[24786]: pgmap v267: 97 pgs: 97 active+clean; 245 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.5 MiB/s wr, 767 op/s 2024-04-14T13:36:33.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:33 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:36:33.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:33 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:36:33.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:33 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:36:34.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:33 smithi012 ceph-mon[27236]: pgmap v267: 97 pgs: 97 active+clean; 245 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.5 MiB/s wr, 767 op/s 2024-04-14T13:36:34.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:33 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:36:34.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:33 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:36:34.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:33 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:36:34.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 42067 17.92 MB/sec warmup 68 sec latency 29.749 ms 2024-04-14T13:36:34.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:34 smithi008 ceph-mon[24786]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (unknown) 2024-04-14T13:36:34.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:34 smithi008 ceph-mon[24786]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T13:36:34.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:34 smithi008 ceph-mon[24786]: Upgrade: Need to upgrade myself (mgr.smithi008.yemlhr) 2024-04-14T13:36:34.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:34 smithi008 ceph-mon[24786]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 on smithi012 2024-04-14T13:36:35.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:34 smithi012 ceph-mon[27236]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (unknown) 2024-04-14T13:36:35.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:34 smithi012 ceph-mon[27236]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T13:36:35.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:34 smithi012 ceph-mon[27236]: Upgrade: Need to upgrade myself (mgr.smithi008.yemlhr) 2024-04-14T13:36:35.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:34 smithi012 ceph-mon[27236]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 on smithi012 2024-04-14T13:36:35.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 42792 18.09 MB/sec warmup 69 sec latency 21.224 ms 2024-04-14T13:36:35.812 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:35 smithi008 ceph-mon[24786]: pgmap v268: 97 pgs: 97 active+clean; 247 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.7 MiB/s wr, 677 op/s 2024-04-14T13:36:36.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:35 smithi012 ceph-mon[27236]: pgmap v268: 97 pgs: 97 active+clean; 247 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.7 MiB/s wr, 677 op/s 2024-04-14T13:36:36.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 43135 17.89 MB/sec warmup 70 sec latency 30.545 ms 2024-04-14T13:36:37.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 43572 17.72 MB/sec warmup 71 sec latency 36.672 ms 2024-04-14T13:36:37.877 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:37 smithi008 ceph-mon[24786]: pgmap v269: 97 pgs: 97 active+clean; 254 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.0 MiB/s wr, 811 op/s 2024-04-14T13:36:38.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:37 smithi012 ceph-mon[27236]: pgmap v269: 97 pgs: 97 active+clean; 254 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.0 MiB/s wr, 811 op/s 2024-04-14T13:36:38.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 44110 17.71 MB/sec warmup 72 sec latency 32.049 ms 2024-04-14T13:36:39.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 44892 17.60 MB/sec warmup 73 sec latency 32.343 ms 2024-04-14T13:36:39.896 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:39 smithi008 ceph-mon[24786]: pgmap v270: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.7 MiB/s wr, 965 op/s 2024-04-14T13:36:40.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:39 smithi012 ceph-mon[27236]: pgmap v270: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.7 MiB/s wr, 965 op/s 2024-04-14T13:36:40.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 45468 17.68 MB/sec warmup 74 sec latency 42.131 ms 2024-04-14T13:36:41.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 46207 17.84 MB/sec warmup 75 sec latency 21.971 ms 2024-04-14T13:36:41.930 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:41 smithi008 ceph-mon[24786]: pgmap v271: 97 pgs: 97 active+clean; 255 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.7 MiB/s wr, 803 op/s 2024-04-14T13:36:42.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:41 smithi012 ceph-mon[27236]: pgmap v271: 97 pgs: 97 active+clean; 255 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.7 MiB/s wr, 803 op/s 2024-04-14T13:36:42.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 46536 17.73 MB/sec warmup 76 sec latency 25.627 ms 2024-04-14T13:36:43.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 47028 17.54 MB/sec warmup 77 sec latency 32.779 ms 2024-04-14T13:36:43.962 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:43 smithi008 ceph-mon[24786]: pgmap v272: 97 pgs: 97 active+clean; 262 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 9.1 MiB/s wr, 764 op/s 2024-04-14T13:36:44.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:43 smithi012 ceph-mon[27236]: pgmap v272: 97 pgs: 97 active+clean; 262 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 9.1 MiB/s wr, 764 op/s 2024-04-14T13:36:44.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 47435 17.49 MB/sec warmup 78 sec latency 63.261 ms 2024-04-14T13:36:45.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 48002 17.40 MB/sec warmup 79 sec latency 62.527 ms 2024-04-14T13:36:45.988 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:45 smithi008 ceph-mon[24786]: pgmap v273: 97 pgs: 97 active+clean; 269 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 827 op/s 2024-04-14T13:36:46.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:45 smithi012 ceph-mon[27236]: pgmap v273: 97 pgs: 97 active+clean; 269 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 827 op/s 2024-04-14T13:36:46.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 48488 17.27 MB/sec warmup 80 sec latency 109.490 ms 2024-04-14T13:36:47.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 48959 17.32 MB/sec warmup 81 sec latency 37.135 ms 2024-04-14T13:36:47.765 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:47 smithi008 ceph-mon[24786]: pgmap v274: 97 pgs: 97 active+clean; 276 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.7 MiB/s wr, 880 op/s 2024-04-14T13:36:48.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:47 smithi012 ceph-mon[27236]: pgmap v274: 97 pgs: 97 active+clean; 276 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.7 MiB/s wr, 880 op/s 2024-04-14T13:36:48.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 49682 17.47 MB/sec warmup 82 sec latency 29.485 ms 2024-04-14T13:36:49.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 50050 17.38 MB/sec warmup 83 sec latency 24.673 ms 2024-04-14T13:36:49.776 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:49 smithi008 ceph-mon[24786]: pgmap v275: 97 pgs: 97 active+clean; 273 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.6 MiB/s wr, 804 op/s 2024-04-14T13:36:50.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:49 smithi012 ceph-mon[27236]: pgmap v275: 97 pgs: 97 active+clean; 273 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.6 MiB/s wr, 804 op/s 2024-04-14T13:36:50.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 50542 17.22 MB/sec warmup 84 sec latency 32.685 ms 2024-04-14T13:36:51.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 51020 17.21 MB/sec warmup 85 sec latency 32.314 ms 2024-04-14T13:36:51.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:51 smithi008 ceph-mon[24786]: pgmap v276: 97 pgs: 97 active+clean; 280 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.2 MiB/s wr, 798 op/s 2024-04-14T13:36:52.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:51 smithi012 ceph-mon[27236]: pgmap v276: 97 pgs: 97 active+clean; 280 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.2 MiB/s wr, 798 op/s 2024-04-14T13:36:52.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 51721 17.11 MB/sec warmup 86 sec latency 38.553 ms 2024-04-14T13:36:53.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 52279 17.07 MB/sec warmup 87 sec latency 24.460 ms 2024-04-14T13:36:53.813 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:53 smithi008 ceph-mon[24786]: pgmap v277: 97 pgs: 97 active+clean; 288 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 7.9 KiB/s rd, 8.6 MiB/s wr, 834 op/s 2024-04-14T13:36:54.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:53 smithi012 ceph-mon[27236]: pgmap v277: 97 pgs: 97 active+clean; 288 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 7.9 KiB/s rd, 8.6 MiB/s wr, 834 op/s 2024-04-14T13:36:54.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 52908 17.19 MB/sec warmup 88 sec latency 36.809 ms 2024-04-14T13:36:55.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 53405 17.25 MB/sec warmup 89 sec latency 75.882 ms 2024-04-14T13:36:55.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:55 smithi008 ceph-mon[24786]: pgmap v278: 97 pgs: 97 active+clean; 285 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.8 MiB/s wr, 670 op/s 2024-04-14T13:36:56.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:55 smithi012 ceph-mon[27236]: pgmap v278: 97 pgs: 97 active+clean; 285 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.8 MiB/s wr, 670 op/s 2024-04-14T13:36:56.304 INFO:teuthology.orchestra.run.smithi008.stdout: 5 53753 17.10 MB/sec warmup 90 sec latency 28.221 ms 2024-04-14T13:36:57.305 INFO:teuthology.orchestra.run.smithi008.stdout: 5 54210 16.97 MB/sec warmup 91 sec latency 41.263 ms 2024-04-14T13:36:57.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:57 smithi008 ceph-mon[24786]: pgmap v279: 97 pgs: 97 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 6.8 MiB/s wr, 677 op/s 2024-04-14T13:36:58.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:57 smithi012 ceph-mon[27236]: pgmap v279: 97 pgs: 97 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 6.8 MiB/s wr, 677 op/s 2024-04-14T13:36:58.305 INFO:teuthology.orchestra.run.smithi008.stdout: 5 54631 16.97 MB/sec warmup 92 sec latency 154.119 ms 2024-04-14T13:36:59.305 INFO:teuthology.orchestra.run.smithi008.stdout: 5 55195 16.84 MB/sec warmup 93 sec latency 75.781 ms 2024-04-14T13:36:59.919 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:36:59 smithi008 ceph-mon[24786]: pgmap v280: 97 pgs: 97 active+clean; 291 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.5 MiB/s wr, 691 op/s 2024-04-14T13:37:00.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:36:59 smithi012 ceph-mon[27236]: pgmap v280: 97 pgs: 97 active+clean; 291 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.5 MiB/s wr, 691 op/s 2024-04-14T13:37:00.305 INFO:teuthology.orchestra.run.smithi008.stdout: 5 55655 16.75 MB/sec warmup 94 sec latency 35.111 ms 2024-04-14T13:37:01.305 INFO:teuthology.orchestra.run.smithi008.stdout: 5 56105 16.80 MB/sec warmup 95 sec latency 37.687 ms 2024-04-14T13:37:01.560 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:37:01.989 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:01 smithi008 ceph-mon[24786]: pgmap v281: 97 pgs: 97 active+clean; 297 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 7.8 MiB/s wr, 702 op/s 2024-04-14T13:37:02.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:01 smithi012 ceph-mon[27236]: pgmap v281: 97 pgs: 97 active+clean; 297 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 7.8 MiB/s wr, 702 op/s 2024-04-14T13:37:02.305 INFO:teuthology.orchestra.run.smithi008.stdout: 5 56840 16.92 MB/sec warmup 96 sec latency 41.700 ms 2024-04-14T13:37:02.548 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:37:02.548 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (6m) 98s ago 8m 23.7M - ba2b418f427c fbc215a6ff5d 2024-04-14T13:37:02.548 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (8m) 98s ago 7m 6405k - 17.2.0 e1d6a67b021e 68b810a9ff93 2024-04-14T13:37:02.548 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (7m) 100s ago 7m 6400k - 17.2.0 e1d6a67b021e 6bc31c51b938 2024-04-14T13:37:02.548 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (6m) 98s ago 7m 54.0M - 8.3.5 dad864ee21e9 c7b42d05cedc 2024-04-14T13:37:02.548 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (2m) 98s ago 2m 14.9M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:37:02.548 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (2m) 100s ago 2m 11.9M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:37:02.549 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:9283 running (9m) 98s ago 9m 460M - 17.2.0 e1d6a67b021e 074f98e540c3 2024-04-14T13:37:02.549 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283 running (7m) 100s ago 7m 412M - 17.2.0 e1d6a67b021e 596e23b54c3d 2024-04-14T13:37:02.549 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (9m) 98s ago 9m 49.7M 2048M 17.2.0 e1d6a67b021e 7786c27a60d0 2024-04-14T13:37:02.549 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (7m) 100s ago 7m 34.5M 2048M 17.2.0 e1d6a67b021e f6cd483dfa31 2024-04-14T13:37:02.549 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (109s) 98s ago 108s 46.9M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:37:02.549 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (103s) 100s ago 103s 46.6M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:37:02.549 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (7m) 98s ago 7m 16.8M - 1dbe0e931976 2338da898186 2024-04-14T13:37:02.549 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (6m) 100s ago 6m 18.7M - 1dbe0e931976 c2fae00595dd 2024-04-14T13:37:02.549 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (6m) 98s ago 6m 43.7M 3217M 17.2.0 e1d6a67b021e 3b8cdea7890c 2024-04-14T13:37:02.549 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (6m) 98s ago 5m 43.8M 3217M 17.2.0 e1d6a67b021e be86642c7790 2024-04-14T13:37:02.549 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (5m) 98s ago 5m 39.6M 3217M 17.2.0 e1d6a67b021e b1017f00578f 2024-04-14T13:37:02.549 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (5m) 98s ago 5m 42.7M 3217M 17.2.0 e1d6a67b021e 66fca4b191be 2024-04-14T13:37:02.549 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (4m) 100s ago 4m 41.4M 3985M 17.2.0 e1d6a67b021e 43b02cb64951 2024-04-14T13:37:02.549 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (4m) 100s ago 4m 38.9M 3985M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:37:02.549 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (3m) 100s ago 3m 39.7M 3985M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:37:02.549 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (3m) 100s ago 3m 42.1M 3985M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:37:02.550 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (6m) 98s ago 6m 62.5M - 514e6a882f6e 11fca64796ed 2024-04-14T13:37:03.001 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:02 smithi008 ceph-mon[24786]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:37:03.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:02 smithi012 ceph-mon[27236]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:37:03.094 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:37:03.094 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:37:03.094 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:37:03.094 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:37:03.094 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:37:03.095 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:37:03.095 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:37:03.095 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:37:03.095 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T13:37:03.095 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:37:03.095 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:37:03.095 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:37:03.095 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:37:03.095 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:37:03.095 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-14T13:37:03.095 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:37:03.095 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:37:03.305 INFO:teuthology.orchestra.run.smithi008.stdout: 5 57254 16.85 MB/sec warmup 97 sec latency 25.384 ms 2024-04-14T13:37:03.606 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:37:03.606 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:37:03.606 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:37:03.606 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-04-14T13:37:03.606 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "0/23 daemons upgraded", 2024-04-14T13:37:03.606 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 image on host smithi012" 2024-04-14T13:37:03.606 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:37:04.008 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:03 smithi008 ceph-mon[24786]: pgmap v282: 97 pgs: 97 active+clean; 299 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.5 MiB/s wr, 673 op/s 2024-04-14T13:37:04.008 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:03 smithi008 ceph-mon[24786]: from='client.14674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:37:04.008 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:03 smithi008 ceph-mon[24786]: from='client.14678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:37:04.008 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:03 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/161879865' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:37:04.049 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:03 smithi012 ceph-mon[27236]: pgmap v282: 97 pgs: 97 active+clean; 299 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.5 MiB/s wr, 673 op/s 2024-04-14T13:37:04.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:03 smithi012 ceph-mon[27236]: from='client.14674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:37:04.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:03 smithi012 ceph-mon[27236]: from='client.14678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:37:04.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:03 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/161879865' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:37:04.216 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-04-14T13:37:04.305 INFO:teuthology.orchestra.run.smithi008.stdout: 5 57772 16.75 MB/sec warmup 98 sec latency 19.414 ms 2024-04-14T13:37:05.014 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:04 smithi008 ceph-mon[24786]: from='client.14686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:37:05.014 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:04 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/1913862722' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:37:05.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:04 smithi012 ceph-mon[27236]: from='client.14686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:37:05.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:04 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/1913862722' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:37:05.305 INFO:teuthology.orchestra.run.smithi008.stdout: 5 58551 16.78 MB/sec warmup 99 sec latency 20.051 ms 2024-04-14T13:37:06.023 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:05 smithi008 ceph-mon[24786]: pgmap v283: 97 pgs: 97 active+clean; 303 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 697 op/s 2024-04-14T13:37:06.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:05 smithi012 ceph-mon[27236]: pgmap v283: 97 pgs: 97 active+clean; 303 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 697 op/s 2024-04-14T13:37:06.305 INFO:teuthology.orchestra.run.smithi008.stdout: 5 59173 16.69 MB/sec warmup 100 sec latency 32.121 ms 2024-04-14T13:37:07.305 INFO:teuthology.orchestra.run.smithi008.stdout: 5 60041 16.88 MB/sec warmup 101 sec latency 24.026 ms 2024-04-14T13:37:07.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:07 smithi008 ceph-mon[24786]: pgmap v284: 97 pgs: 97 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 718 op/s 2024-04-14T13:37:08.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:07 smithi012 ceph-mon[27236]: pgmap v284: 97 pgs: 97 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 718 op/s 2024-04-14T13:37:08.305 INFO:teuthology.orchestra.run.smithi008.stdout: 5 60705 16.96 MB/sec warmup 102 sec latency 22.344 ms 2024-04-14T13:37:09.305 INFO:teuthology.orchestra.run.smithi008.stdout: 5 61306 16.85 MB/sec warmup 103 sec latency 19.127 ms 2024-04-14T13:37:10.050 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:09 smithi008 ceph-mon[24786]: pgmap v285: 97 pgs: 97 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.2 KiB/s rd, 8.1 MiB/s wr, 723 op/s 2024-04-14T13:37:10.050 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:09 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:37:10.050 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:09 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi012.ksvnxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:37:10.050 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:09 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:37:10.050 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:09 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:37:10.276 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:09 smithi012 ceph-mon[27236]: pgmap v285: 97 pgs: 97 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.2 KiB/s rd, 8.1 MiB/s wr, 723 op/s 2024-04-14T13:37:10.276 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:09 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:37:10.276 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:09 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi012.ksvnxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:37:10.276 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:09 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:37:10.276 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:09 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:37:10.305 INFO:teuthology.orchestra.run.smithi008.stdout: 5 62012 16.89 MB/sec warmup 104 sec latency 28.722 ms 2024-04-14T13:37:11.095 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:10 smithi008 ceph-mon[24786]: Upgrade: Updating mgr.smithi012.ksvnxm 2024-04-14T13:37:11.095 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:10 smithi008 ceph-mon[24786]: Deploying daemon mgr.smithi012.ksvnxm on smithi012 2024-04-14T13:37:11.252 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:10 smithi012 ceph-mon[27236]: Upgrade: Updating mgr.smithi012.ksvnxm 2024-04-14T13:37:11.252 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:10 smithi012 ceph-mon[27236]: Deploying daemon mgr.smithi012.ksvnxm on smithi012 2024-04-14T13:37:11.306 INFO:teuthology.orchestra.run.smithi008.stdout: 5 62664 16.80 MB/sec warmup 105 sec latency 92.300 ms 2024-04-14T13:37:11.871 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:11 smithi008 ceph-mon[24786]: pgmap v286: 97 pgs: 97 active+clean; 320 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.8 MiB/s wr, 890 op/s 2024-04-14T13:37:12.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:11 smithi012 ceph-mon[27236]: pgmap v286: 97 pgs: 97 active+clean; 320 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.8 MiB/s wr, 890 op/s 2024-04-14T13:37:12.306 INFO:teuthology.orchestra.run.smithi008.stdout: 5 63075 16.82 MB/sec warmup 106 sec latency 56.430 ms 2024-04-14T13:37:13.306 INFO:teuthology.orchestra.run.smithi008.stdout: 5 63695 16.86 MB/sec warmup 107 sec latency 51.740 ms 2024-04-14T13:37:13.949 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:13 smithi008 ceph-mon[24786]: pgmap v287: 97 pgs: 97 active+clean; 318 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 8.6 KiB/s rd, 9.0 MiB/s wr, 821 op/s 2024-04-14T13:37:14.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:13 smithi012 ceph-mon[27236]: pgmap v287: 97 pgs: 97 active+clean; 318 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 8.6 KiB/s rd, 9.0 MiB/s wr, 821 op/s 2024-04-14T13:37:14.306 INFO:teuthology.orchestra.run.smithi008.stdout: 5 64098 16.88 MB/sec warmup 108 sec latency 84.744 ms 2024-04-14T13:37:15.306 INFO:teuthology.orchestra.run.smithi008.stdout: 5 64446 16.75 MB/sec warmup 109 sec latency 36.119 ms 2024-04-14T13:37:16.049 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:15 smithi008 ceph-mon[24786]: pgmap v288: 97 pgs: 97 active+clean; 324 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 8.9 MiB/s wr, 915 op/s 2024-04-14T13:37:16.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:15 smithi012 ceph-mon[27236]: pgmap v288: 97 pgs: 97 active+clean; 324 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 8.9 MiB/s wr, 915 op/s 2024-04-14T13:37:16.306 INFO:teuthology.orchestra.run.smithi008.stdout: 5 64889 16.65 MB/sec warmup 110 sec latency 65.757 ms 2024-04-14T13:37:17.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 65373 16.68 MB/sec warmup 111 sec latency 72.982 ms 2024-04-14T13:37:18.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:17 smithi012 ceph-mon[27236]: pgmap v289: 97 pgs: 97 active+clean; 326 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.4 MiB/s wr, 829 op/s 2024-04-14T13:37:18.061 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:17 smithi008 ceph-mon[24786]: pgmap v289: 97 pgs: 97 active+clean; 326 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.4 MiB/s wr, 829 op/s 2024-04-14T13:37:18.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 66027 16.57 MB/sec warmup 112 sec latency 47.570 ms 2024-04-14T13:37:18.887 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:18 smithi008 ceph-mon[24786]: pgmap v290: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 9.1 MiB/s wr, 832 op/s 2024-04-14T13:37:19.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:18 smithi012 ceph-mon[27236]: pgmap v290: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 9.1 MiB/s wr, 832 op/s 2024-04-14T13:37:19.306 INFO:teuthology.orchestra.run.smithi008.stdout: 5 66441 16.53 MB/sec warmup 113 sec latency 43.307 ms 2024-04-14T13:37:20.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 66964 16.59 MB/sec warmup 114 sec latency 73.529 ms 2024-04-14T13:37:21.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 67497 16.65 MB/sec warmup 115 sec latency 48.676 ms 2024-04-14T13:37:21.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:21 smithi008 ceph-mon[24786]: pgmap v291: 97 pgs: 97 active+clean; 335 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 8.7 MiB/s wr, 932 op/s 2024-04-14T13:37:21.819 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:21 smithi012 ceph-mon[27236]: pgmap v291: 97 pgs: 97 active+clean; 335 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 8.7 MiB/s wr, 932 op/s 2024-04-14T13:37:22.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 67761 16.55 MB/sec warmup 116 sec latency 48.327 ms 2024-04-14T13:37:23.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 68062 16.43 MB/sec warmup 117 sec latency 82.623 ms 2024-04-14T13:37:23.797 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:23 smithi012 ceph-mon[27236]: pgmap v292: 97 pgs: 97 active+clean; 335 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.2 MiB/s wr, 837 op/s 2024-04-14T13:37:23.797 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:23 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:37:23.797 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:23 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:37:23.797 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:23 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:37:23.797 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:23 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:37:23.944 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:23 smithi008 ceph-mon[24786]: pgmap v292: 97 pgs: 97 active+clean; 335 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.2 MiB/s wr, 837 op/s 2024-04-14T13:37:23.944 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:23 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:37:23.944 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:23 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:37:23.944 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:23 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:37:23.944 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:23 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:37:24.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 68598 16.39 MB/sec warmup 118 sec latency 26.265 ms 2024-04-14T13:37:25.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 69312 16.37 MB/sec warmup 119 sec latency 31.185 ms 2024-04-14T13:37:25.703 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:25 smithi008 ceph-mon[24786]: pgmap v293: 97 pgs: 97 active+clean; 334 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.9 KiB/s rd, 7.1 MiB/s wr, 788 op/s 2024-04-14T13:37:25.799 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:25 smithi012 ceph-mon[27236]: pgmap v293: 97 pgs: 97 active+clean; 334 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.9 KiB/s rd, 7.1 MiB/s wr, 788 op/s 2024-04-14T13:37:27.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 70444 24.77 MB/sec execute 1 sec latency 34.491 ms 2024-04-14T13:37:27.468 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:27 smithi008 ceph-mon[24786]: pgmap v294: 97 pgs: 97 active+clean; 340 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 6.5 MiB/s wr, 637 op/s 2024-04-14T13:37:27.468 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:27 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:37:27.468 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:27 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:37:27.617 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:27 smithi012 ceph-mon[27236]: pgmap v294: 97 pgs: 97 active+clean; 340 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 6.5 MiB/s wr, 637 op/s 2024-04-14T13:37:27.618 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:27 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:37:27.618 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:27 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:37:28.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 71046 24.86 MB/sec execute 2 sec latency 51.682 ms 2024-04-14T13:37:29.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 71362 18.50 MB/sec execute 3 sec latency 38.002 ms 2024-04-14T13:37:29.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:29 smithi008 ceph-mon[24786]: pgmap v295: 97 pgs: 97 active+clean; 329 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 7.4 MiB/s wr, 690 op/s 2024-04-14T13:37:29.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:29 smithi012 ceph-mon[27236]: pgmap v295: 97 pgs: 97 active+clean; 329 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 7.4 MiB/s wr, 690 op/s 2024-04-14T13:37:30.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 71736 14.61 MB/sec execute 4 sec latency 35.252 ms 2024-04-14T13:37:31.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 72080 13.40 MB/sec execute 5 sec latency 33.248 ms 2024-04-14T13:37:31.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:31 smithi008 ceph-mon[24786]: pgmap v296: 97 pgs: 97 active+clean; 331 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.5 MiB/s wr, 815 op/s 2024-04-14T13:37:31.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:31 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:37:31.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:31 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:37:31.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:31 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:37:31.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:31 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:37:31.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:31 smithi008 ceph-mon[24786]: Upgrade: Need to upgrade myself (mgr.smithi008.yemlhr) 2024-04-14T13:37:31.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:31 smithi008 ceph-mon[24786]: Failing over to other MGR 2024-04-14T13:37:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:31 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr fail", "who": "smithi008.yemlhr"}]: dispatch 2024-04-14T13:37:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:31 smithi008 ceph-mon[24786]: osdmap e70: 8 total, 8 up, 8 in 2024-04-14T13:37:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:31 smithi008 ceph-mon[24786]: Active manager daemon smithi012.ksvnxm restarted 2024-04-14T13:37:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:31 smithi008 ceph-mon[24786]: Activating manager daemon smithi012.ksvnxm 2024-04-14T13:37:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:31 smithi008 ceph-mon[24786]: osdmap e71: 8 total, 8 up, 8 in 2024-04-14T13:37:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:31 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi012.ksvnxm/crt"}]: dispatch 2024-04-14T13:37:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:31 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T13:37:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:31 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi012.ksvnxm/key"}]: dispatch 2024-04-14T13:37:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:31 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T13:37:32.007 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:31 smithi012 ceph-mon[27236]: pgmap v296: 97 pgs: 97 active+clean; 331 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.5 MiB/s wr, 815 op/s 2024-04-14T13:37:32.007 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:31 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:37:32.007 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:31 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:37:32.007 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:31 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' 2024-04-14T13:37:32.008 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:31 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:37:32.008 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:31 smithi012 ceph-mon[27236]: Upgrade: Need to upgrade myself (mgr.smithi008.yemlhr) 2024-04-14T13:37:32.008 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:31 smithi012 ceph-mon[27236]: Failing over to other MGR 2024-04-14T13:37:32.008 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:31 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr fail", "who": "smithi008.yemlhr"}]: dispatch 2024-04-14T13:37:32.008 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:31 smithi012 ceph-mon[27236]: osdmap e70: 8 total, 8 up, 8 in 2024-04-14T13:37:32.008 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:31 smithi012 ceph-mon[27236]: Active manager daemon smithi012.ksvnxm restarted 2024-04-14T13:37:32.008 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:31 smithi012 ceph-mon[27236]: Activating manager daemon smithi012.ksvnxm 2024-04-14T13:37:32.008 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:31 smithi012 ceph-mon[27236]: osdmap e71: 8 total, 8 up, 8 in 2024-04-14T13:37:32.008 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:31 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi012.ksvnxm/crt"}]: dispatch 2024-04-14T13:37:32.008 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:31 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T13:37:32.008 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:31 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi012.ksvnxm/key"}]: dispatch 2024-04-14T13:37:32.008 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:31 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T13:37:32.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 72501 13.21 MB/sec execute 6 sec latency 49.743 ms 2024-04-14T13:37:32.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "mgr fail", "who": "smithi008.yemlhr"}]': finished 2024-04-14T13:37:32.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: mgrmap e22: smithi012.ksvnxm(active, starting, since 0.47781s) 2024-04-14T13:37:32.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:37:32.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:37:32.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wqbvfi"}]: dispatch 2024-04-14T13:37:32.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi012.vjovzp"}]: dispatch 2024-04-14T13:37:32.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mgr metadata", "who": "smithi012.ksvnxm", "id": "smithi012.ksvnxm"}]: dispatch 2024-04-14T13:37:32.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:37:32.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:37:32.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:37:32.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:37:32.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:37:32.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:37:32.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:37:32.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:37:32.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T13:37:32.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T13:37:32.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T13:37:32.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: Manager daemon smithi012.ksvnxm is now available 2024-04-14T13:37:32.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:32.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:37:32.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:37:32.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi012.ksvnxm/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:37:32.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi012.ksvnxm/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:37:32.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi012.ksvnxm/trash_purge_schedule"}]: dispatch 2024-04-14T13:37:32.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi012.ksvnxm/trash_purge_schedule"}]: dispatch 2024-04-14T13:37:32.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: Deploying cephadm binary to smithi012 2024-04-14T13:37:32.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:32 smithi008 ceph-mon[24786]: Deploying cephadm binary to smithi008 2024-04-14T13:37:33.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.14206 172.21.15.8:0/3868621691' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "mgr fail", "who": "smithi008.yemlhr"}]': finished 2024-04-14T13:37:33.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: mgrmap e22: smithi012.ksvnxm(active, starting, since 0.47781s) 2024-04-14T13:37:33.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:37:33.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:37:33.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wqbvfi"}]: dispatch 2024-04-14T13:37:33.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi012.vjovzp"}]: dispatch 2024-04-14T13:37:33.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mgr metadata", "who": "smithi012.ksvnxm", "id": "smithi012.ksvnxm"}]: dispatch 2024-04-14T13:37:33.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:37:33.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:37:33.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:37:33.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:37:33.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:37:33.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:37:33.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:37:33.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:37:33.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T13:37:33.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T13:37:33.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T13:37:33.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: Manager daemon smithi012.ksvnxm is now available 2024-04-14T13:37:33.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:33.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:37:33.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:37:33.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi012.ksvnxm/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:37:33.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi012.ksvnxm/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:37:33.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi012.ksvnxm/trash_purge_schedule"}]: dispatch 2024-04-14T13:37:33.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi012.ksvnxm/trash_purge_schedule"}]: dispatch 2024-04-14T13:37:33.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: Deploying cephadm binary to smithi012 2024-04-14T13:37:33.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:32 smithi012 ceph-mon[27236]: Deploying cephadm binary to smithi008 2024-04-14T13:37:33.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 73132 12.39 MB/sec execute 7 sec latency 42.660 ms 2024-04-14T13:37:33.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:33 smithi008 ceph-mon[24786]: mgrmap e23: smithi012.ksvnxm(active, since 1.48874s) 2024-04-14T13:37:33.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:33 smithi008 ceph-mon[24786]: pgmap v3: 97 pgs: 97 active+clean; 317 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-14T13:37:33.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:33 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:33.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:33 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:34.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:33 smithi012 ceph-mon[27236]: mgrmap e23: smithi012.ksvnxm(active, since 1.48874s) 2024-04-14T13:37:34.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:33 smithi012 ceph-mon[27236]: pgmap v3: 97 pgs: 97 active+clean; 317 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-14T13:37:34.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:33 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:34.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:33 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:34.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 73746 13.22 MB/sec execute 8 sec latency 16.881 ms 2024-04-14T13:37:34.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:34 smithi008 ceph-mon[24786]: pgmap v4: 97 pgs: 97 active+clean; 317 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-14T13:37:34.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:34 smithi008 ceph-mon[24786]: mgrmap e24: smithi012.ksvnxm(active, since 2s) 2024-04-14T13:37:34.878 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:37:35.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:34 smithi012 ceph-mon[27236]: pgmap v4: 97 pgs: 97 active+clean; 317 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-14T13:37:35.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:34 smithi012 ceph-mon[27236]: mgrmap e24: smithi012.ksvnxm(active, since 2s) 2024-04-14T13:37:35.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 74330 14.47 MB/sec execute 9 sec latency 22.890 ms 2024-04-14T13:37:35.810 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:35 smithi008 ceph-mon[24786]: from='client.24497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:37:35.810 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:35 smithi008 ceph-mon[24786]: from='client.14716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:37:36.094 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:37:36.094 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (6m) 2m ago 8m 23.7M - ba2b418f427c fbc215a6ff5d 2024-04-14T13:37:36.094 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (8m) 2m ago 8m 6405k - 17.2.0 e1d6a67b021e 68b810a9ff93 2024-04-14T13:37:36.094 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (7m) 9s ago 7m 6400k - 17.2.0 e1d6a67b021e 6bc31c51b938 2024-04-14T13:37:36.094 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (6m) 2m ago 8m 54.0M - 8.3.5 dad864ee21e9 c7b42d05cedc 2024-04-14T13:37:36.095 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (2m) 2m ago 2m 14.9M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:37:36.095 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (2m) 9s ago 2m 12.3M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:37:36.095 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:9283 running (9m) 2m ago 9m 460M - 17.2.0 e1d6a67b021e 074f98e540c3 2024-04-14T13:37:36.095 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283 running (13s) 9s ago 7m 51.3M - 19.0.0-2778-ga971fd5f b96c509eef84 089c1c1b8529 2024-04-14T13:37:36.095 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (10m) 2m ago 10m 49.7M 2048M 17.2.0 e1d6a67b021e 7786c27a60d0 2024-04-14T13:37:36.095 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (7m) 9s ago 7m 37.4M 2048M 17.2.0 e1d6a67b021e f6cd483dfa31 2024-04-14T13:37:36.095 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (2m) 2m ago 2m 46.9M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:37:36.095 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (2m) 9s ago 2m 73.7M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:37:36.095 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (8m) 2m ago 8m 16.8M - 1dbe0e931976 2338da898186 2024-04-14T13:37:36.095 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (7m) 9s ago 7m 18.4M - 1dbe0e931976 c2fae00595dd 2024-04-14T13:37:36.095 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (7m) 2m ago 7m 43.7M 3217M 17.2.0 e1d6a67b021e 3b8cdea7890c 2024-04-14T13:37:36.095 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (6m) 2m ago 6m 43.8M 3217M 17.2.0 e1d6a67b021e be86642c7790 2024-04-14T13:37:36.095 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (6m) 2m ago 6m 39.6M 3217M 17.2.0 e1d6a67b021e b1017f00578f 2024-04-14T13:37:36.095 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (5m) 2m ago 5m 42.7M 3217M 17.2.0 e1d6a67b021e 66fca4b191be 2024-04-14T13:37:36.095 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (5m) 9s ago 5m 262M 3985M 17.2.0 e1d6a67b021e 43b02cb64951 2024-04-14T13:37:36.095 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (4m) 9s ago 4m 292M 3985M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:37:36.096 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (4m) 9s ago 4m 263M 3985M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:37:36.096 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (4m) 9s ago 4m 367M 3985M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:37:36.096 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (7m) 2m ago 7m 62.5M - 514e6a882f6e 11fca64796ed 2024-04-14T13:37:36.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:35 smithi012 ceph-mon[27236]: from='client.24497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:37:36.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:35 smithi012 ceph-mon[27236]: from='client.14716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:37:36.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 74709 14.70 MB/sec execute 10 sec latency 39.660 ms 2024-04-14T13:37:36.764 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:37:36.764 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:37:36.764 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:37:36.764 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:37:36.764 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:37:36.764 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T13:37:36.764 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:37:36.764 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:37:36.764 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T13:37:36.765 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:37:36.765 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:37:36.765 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:37:36.765 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:37:36.765 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:37:36.765 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-14T13:37:36.765 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T13:37:36.765 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:37:36.765 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:37:36.825 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:36 smithi008 ceph-mon[24786]: [14/Apr/2024:13:37:35] ENGINE Bus STARTING 2024-04-14T13:37:36.825 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:36 smithi008 ceph-mon[24786]: pgmap v5: 97 pgs: 97 active+clean; 317 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-14T13:37:36.825 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:36 smithi008 ceph-mon[24786]: mgrmap e25: smithi012.ksvnxm(active, since 4s) 2024-04-14T13:37:36.825 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:36 smithi008 ceph-mon[24786]: [14/Apr/2024:13:37:35] ENGINE Serving on http://172.21.15.12:8765 2024-04-14T13:37:36.825 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:36 smithi008 ceph-mon[24786]: [14/Apr/2024:13:37:35] ENGINE Serving on https://172.21.15.12:7150 2024-04-14T13:37:36.825 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:36 smithi008 ceph-mon[24786]: [14/Apr/2024:13:37:35] ENGINE Bus STARTED 2024-04-14T13:37:36.825 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:36 smithi008 ceph-mon[24786]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:37:36.825 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:36 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2347514522' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:37:37.220 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:36 smithi012 ceph-mon[27236]: [14/Apr/2024:13:37:35] ENGINE Bus STARTING 2024-04-14T13:37:37.220 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:36 smithi012 ceph-mon[27236]: pgmap v5: 97 pgs: 97 active+clean; 317 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-14T13:37:37.220 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:36 smithi012 ceph-mon[27236]: mgrmap e25: smithi012.ksvnxm(active, since 4s) 2024-04-14T13:37:37.221 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:36 smithi012 ceph-mon[27236]: [14/Apr/2024:13:37:35] ENGINE Serving on http://172.21.15.12:8765 2024-04-14T13:37:37.221 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:36 smithi012 ceph-mon[27236]: [14/Apr/2024:13:37:35] ENGINE Serving on https://172.21.15.12:7150 2024-04-14T13:37:37.221 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:36 smithi012 ceph-mon[27236]: [14/Apr/2024:13:37:35] ENGINE Bus STARTED 2024-04-14T13:37:37.221 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:36 smithi012 ceph-mon[27236]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:37:37.221 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:36 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/2347514522' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:37:37.306 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:37:37.306 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:37:37.306 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:37:37.306 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:37:37.306 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-04-14T13:37:37.306 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "1/23 daemons upgraded", 2024-04-14T13:37:37.307 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "", 2024-04-14T13:37:37.307 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:37:37.307 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:37:37.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 74942 13.78 MB/sec execute 11 sec latency 67.641 ms 2024-04-14T13:37:37.839 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:37 smithi008 ceph-mon[24786]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:37:37.994 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-04-14T13:37:38.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:37 smithi012 ceph-mon[27236]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:37:38.307 INFO:teuthology.orchestra.run.smithi008.stdout: 5 75265 12.90 MB/sec execute 12 sec latency 31.184 ms 2024-04-14T13:37:38.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:38 smithi008 ceph-mon[24786]: pgmap v6: 97 pgs: 97 active+clean; 317 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-14T13:37:38.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:38 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/4286912452' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:37:39.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:38 smithi012 ceph-mon[27236]: pgmap v6: 97 pgs: 97 active+clean; 317 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-14T13:37:39.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:38 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/4286912452' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:37:39.308 INFO:teuthology.orchestra.run.smithi008.stdout: 5 75627 12.42 MB/sec execute 13 sec latency 71.637 ms 2024-04-14T13:37:40.308 INFO:teuthology.orchestra.run.smithi008.stdout: 5 76136 12.38 MB/sec execute 14 sec latency 51.483 ms 2024-04-14T13:37:40.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:40 smithi008 ceph-mon[24786]: pgmap v7: 97 pgs: 97 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 34 KiB/s rd, 5.4 MiB/s wr, 466 op/s 2024-04-14T13:37:41.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:40 smithi012 ceph-mon[27236]: pgmap v7: 97 pgs: 97 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 34 KiB/s rd, 5.4 MiB/s wr, 466 op/s 2024-04-14T13:37:41.308 INFO:teuthology.orchestra.run.smithi008.stdout: 5 76800 12.48 MB/sec execute 15 sec latency 33.550 ms 2024-04-14T13:37:41.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:41 smithi008 ceph-mon[24786]: Standby manager daemon smithi008.yemlhr started 2024-04-14T13:37:41.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:41 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.8:0/926775817' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.yemlhr/crt"}]: dispatch 2024-04-14T13:37:41.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:41 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.8:0/926775817' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T13:37:41.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:41 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.8:0/926775817' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.yemlhr/key"}]: dispatch 2024-04-14T13:37:41.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:41 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.8:0/926775817' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T13:37:42.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:41 smithi012 ceph-mon[27236]: Standby manager daemon smithi008.yemlhr started 2024-04-14T13:37:42.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:41 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.8:0/926775817' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.yemlhr/crt"}]: dispatch 2024-04-14T13:37:42.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:41 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.8:0/926775817' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T13:37:42.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:41 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.8:0/926775817' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.yemlhr/key"}]: dispatch 2024-04-14T13:37:42.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:41 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.8:0/926775817' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T13:37:42.308 INFO:teuthology.orchestra.run.smithi008.stdout: 5 77323 12.75 MB/sec execute 16 sec latency 38.665 ms 2024-04-14T13:37:42.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:42 smithi008 ceph-mon[24786]: pgmap v8: 97 pgs: 97 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 4.2 MiB/s wr, 362 op/s 2024-04-14T13:37:42.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:42 smithi008 ceph-mon[24786]: mgrmap e26: smithi012.ksvnxm(active, since 10s), standbys: smithi008.yemlhr 2024-04-14T13:37:42.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:42 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mgr metadata", "who": "smithi008.yemlhr", "id": "smithi008.yemlhr"}]: dispatch 2024-04-14T13:37:43.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:42 smithi012 ceph-mon[27236]: pgmap v8: 97 pgs: 97 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 4.2 MiB/s wr, 362 op/s 2024-04-14T13:37:43.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:42 smithi012 ceph-mon[27236]: mgrmap e26: smithi012.ksvnxm(active, since 10s), standbys: smithi008.yemlhr 2024-04-14T13:37:43.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:42 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mgr metadata", "who": "smithi008.yemlhr", "id": "smithi008.yemlhr"}]: dispatch 2024-04-14T13:37:43.308 INFO:teuthology.orchestra.run.smithi008.stdout: 5 77931 13.32 MB/sec execute 17 sec latency 25.191 ms 2024-04-14T13:37:44.308 INFO:teuthology.orchestra.run.smithi008.stdout: 5 78459 13.73 MB/sec execute 18 sec latency 26.288 ms 2024-04-14T13:37:44.902 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:44 smithi008 ceph-mon[24786]: pgmap v9: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 6.5 MiB/s wr, 628 op/s 2024-04-14T13:37:45.167 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:44 smithi012 ceph-mon[27236]: pgmap v9: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 6.5 MiB/s wr, 628 op/s 2024-04-14T13:37:45.308 INFO:teuthology.orchestra.run.smithi008.stdout: 5 78862 13.20 MB/sec execute 19 sec latency 26.099 ms 2024-04-14T13:37:46.308 INFO:teuthology.orchestra.run.smithi008.stdout: 5 79263 12.93 MB/sec execute 20 sec latency 65.995 ms 2024-04-14T13:37:46.429 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:46 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:46.429 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:46 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:46.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:46 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:46.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:46 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:47.308 INFO:teuthology.orchestra.run.smithi008.stdout: 5 79711 12.82 MB/sec execute 21 sec latency 35.460 ms 2024-04-14T13:37:47.460 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:47 smithi008 ceph-mon[24786]: pgmap v10: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.9 MiB/s wr, 573 op/s 2024-04-14T13:37:47.460 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:47 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:37:47.460 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:47 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:47.460 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:47 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:47.460 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:47 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:47.460 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:47 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:47.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:47 smithi012 ceph-mon[27236]: pgmap v10: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.9 MiB/s wr, 573 op/s 2024-04-14T13:37:47.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:47 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:37:47.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:47 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:47.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:47 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:47.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:47 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:47.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:47 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:48.308 INFO:teuthology.orchestra.run.smithi008.stdout: 5 80355 12.89 MB/sec execute 22 sec latency 39.764 ms 2024-04-14T13:37:49.308 INFO:teuthology.orchestra.run.smithi008.stdout: 5 80960 13.24 MB/sec execute 23 sec latency 22.566 ms 2024-04-14T13:37:49.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:49 smithi008 ceph-mon[24786]: pgmap v11: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.9 MiB/s wr, 573 op/s 2024-04-14T13:37:49.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:49 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:49.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:49 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:49.733 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:49 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:49.733 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:49 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:49.733 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:49 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:49.733 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:49 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:49.733 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:49 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:49.733 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:49 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:49.733 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:49 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:49.733 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:49 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:49.734 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:49 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:49.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:49 smithi012 ceph-mon[27236]: pgmap v11: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.9 MiB/s wr, 573 op/s 2024-04-14T13:37:49.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:49 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:49.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:49 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:49.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:49 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:49.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:49 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:49.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:49 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:49.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:49 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:49.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:49 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:49.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:49 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:49.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:49 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:49.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:49 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:49.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:49 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:50.308 INFO:teuthology.orchestra.run.smithi008.stdout: 5 81615 13.72 MB/sec execute 24 sec latency 33.675 ms 2024-04-14T13:37:50.746 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:50 smithi008 ceph-mon[24786]: Adjusting osd_memory_target on smithi008 to 1937M 2024-04-14T13:37:50.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:50 smithi012 ceph-mon[27236]: Adjusting osd_memory_target on smithi008 to 1937M 2024-04-14T13:37:51.308 INFO:teuthology.orchestra.run.smithi008.stdout: 5 82047 13.73 MB/sec execute 25 sec latency 28.350 ms 2024-04-14T13:37:51.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:51 smithi008 ceph-mon[24786]: pgmap v12: 97 pgs: 97 active+clean; 323 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 8.8 MiB/s wr, 866 op/s 2024-04-14T13:37:51.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:51 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:51.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:51 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:51.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:51 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:51.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:51 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:51.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:51 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:51.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:51 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:51.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:51 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:51.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:51 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:51.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:51 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:51.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:51 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:51.761 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:51 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:51.761 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:51 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:37:51.761 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:51 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:37:52.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:51 smithi012 ceph-mon[27236]: pgmap v12: 97 pgs: 97 active+clean; 323 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 8.8 MiB/s wr, 866 op/s 2024-04-14T13:37:52.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:51 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:52.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:51 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:52.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:51 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:52.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:51 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:52.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:51 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:52.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:51 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:52.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:51 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:52.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:51 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:52.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:51 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:52.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:51 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:37:52.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:51 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:52.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:51 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:37:52.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:51 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:37:52.308 INFO:teuthology.orchestra.run.smithi008.stdout: 5 82559 13.47 MB/sec execute 26 sec latency 36.480 ms 2024-04-14T13:37:52.771 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: Adjusting osd_memory_target on smithi012 to 2705M 2024-04-14T13:37:52.771 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: Updating smithi008:/etc/ceph/ceph.conf 2024-04-14T13:37:52.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: Updating smithi012:/etc/ceph/ceph.conf 2024-04-14T13:37:52.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: Updating smithi012:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T13:37:52.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: Updating smithi008:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T13:37:52.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:37:52.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:37:52.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: Updating smithi012:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T13:37:52.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: pgmap v13: 97 pgs: 97 active+clean; 323 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.6 MiB/s wr, 596 op/s 2024-04-14T13:37:52.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: Updating smithi008:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T13:37:52.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:52.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:52.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:52.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:52.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:52.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:52.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:52.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:52.773 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:52.773 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:52.773 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: Reconfiguring alertmanager.smithi008 (dependencies changed)... 2024-04-14T13:37:52.773 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:52 smithi008 ceph-mon[24786]: Reconfiguring daemon alertmanager.smithi008 on smithi008 2024-04-14T13:37:53.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: Adjusting osd_memory_target on smithi012 to 2705M 2024-04-14T13:37:53.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: Updating smithi008:/etc/ceph/ceph.conf 2024-04-14T13:37:53.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: Updating smithi012:/etc/ceph/ceph.conf 2024-04-14T13:37:53.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: Updating smithi012:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T13:37:53.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: Updating smithi008:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T13:37:53.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:37:53.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:37:53.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: Updating smithi012:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T13:37:53.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: pgmap v13: 97 pgs: 97 active+clean; 323 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.6 MiB/s wr, 596 op/s 2024-04-14T13:37:53.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: Updating smithi008:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T13:37:53.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:53.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:53.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:53.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:53.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:53.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:53.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:53.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:53.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:53.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:37:53.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: Reconfiguring alertmanager.smithi008 (dependencies changed)... 2024-04-14T13:37:53.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:52 smithi012 ceph-mon[27236]: Reconfiguring daemon alertmanager.smithi008 on smithi008 2024-04-14T13:37:53.308 INFO:teuthology.orchestra.run.smithi008.stdout: 5 83109 13.45 MB/sec execute 27 sec latency 35.006 ms 2024-04-14T13:37:54.308 INFO:teuthology.orchestra.run.smithi008.stdout: 5 83764 13.45 MB/sec execute 28 sec latency 34.809 ms 2024-04-14T13:37:55.039 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:54 smithi008 ceph-mon[24786]: pgmap v14: 97 pgs: 97 active+clean; 332 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.4 MiB/s wr, 934 op/s 2024-04-14T13:37:55.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:54 smithi012 ceph-mon[27236]: pgmap v14: 97 pgs: 97 active+clean; 332 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.4 MiB/s wr, 934 op/s 2024-04-14T13:37:55.308 INFO:teuthology.orchestra.run.smithi008.stdout: 5 84312 13.47 MB/sec execute 29 sec latency 31.669 ms 2024-04-14T13:37:56.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 84872 13.83 MB/sec execute 30 sec latency 28.749 ms 2024-04-14T13:37:56.828 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:56 smithi008 ceph-mon[24786]: pgmap v15: 97 pgs: 97 active+clean; 332 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 6.6 MiB/s wr, 630 op/s 2024-04-14T13:37:57.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:56 smithi012 ceph-mon[27236]: pgmap v15: 97 pgs: 97 active+clean; 332 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 6.6 MiB/s wr, 630 op/s 2024-04-14T13:37:57.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 85412 14.04 MB/sec execute 31 sec latency 41.051 ms 2024-04-14T13:37:58.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 85871 13.80 MB/sec execute 32 sec latency 23.058 ms 2024-04-14T13:37:58.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:37:58 smithi008 ceph-mon[24786]: pgmap v16: 97 pgs: 97 active+clean; 332 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 6.6 MiB/s wr, 630 op/s 2024-04-14T13:37:59.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:37:58 smithi012 ceph-mon[27236]: pgmap v16: 97 pgs: 97 active+clean; 332 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 6.6 MiB/s wr, 630 op/s 2024-04-14T13:37:59.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 86449 13.65 MB/sec execute 33 sec latency 28.994 ms 2024-04-14T13:38:00.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 87023 13.76 MB/sec execute 34 sec latency 25.813 ms 2024-04-14T13:38:00.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:00 smithi008 ceph-mon[24786]: pgmap v17: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 956 op/s 2024-04-14T13:38:01.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:00 smithi012 ceph-mon[27236]: pgmap v17: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 956 op/s 2024-04-14T13:38:01.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 87559 13.63 MB/sec execute 35 sec latency 37.277 ms 2024-04-14T13:38:02.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 88049 13.73 MB/sec execute 36 sec latency 31.861 ms 2024-04-14T13:38:02.876 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:02 smithi008 ceph-mon[24786]: pgmap v18: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.3 MiB/s wr, 662 op/s 2024-04-14T13:38:02.876 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:02 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:38:03.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:02 smithi012 ceph-mon[27236]: pgmap v18: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.3 MiB/s wr, 662 op/s 2024-04-14T13:38:03.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:02 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:38:03.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 88466 13.82 MB/sec execute 37 sec latency 25.608 ms 2024-04-14T13:38:04.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 88888 13.88 MB/sec execute 38 sec latency 33.274 ms 2024-04-14T13:38:04.837 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:04 smithi008 ceph-mon[24786]: pgmap v19: 97 pgs: 97 active+clean; 325 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 972 op/s 2024-04-14T13:38:05.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:04 smithi012 ceph-mon[27236]: pgmap v19: 97 pgs: 97 active+clean; 325 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 972 op/s 2024-04-14T13:38:05.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 89351 13.76 MB/sec execute 39 sec latency 35.854 ms 2024-04-14T13:38:06.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 89936 13.64 MB/sec execute 40 sec latency 23.108 ms 2024-04-14T13:38:06.590 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:06 smithi008 ceph-mon[24786]: pgmap v20: 97 pgs: 97 active+clean; 325 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.5 MiB/s wr, 634 op/s 2024-04-14T13:38:07.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:06 smithi012 ceph-mon[27236]: pgmap v20: 97 pgs: 97 active+clean; 325 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.5 MiB/s wr, 634 op/s 2024-04-14T13:38:07.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 90518 13.74 MB/sec execute 41 sec latency 22.307 ms 2024-04-14T13:38:08.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 91175 13.66 MB/sec execute 42 sec latency 46.884 ms 2024-04-14T13:38:08.612 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:38:08.866 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:08 smithi008 ceph-mon[24786]: pgmap v21: 97 pgs: 97 active+clean; 325 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.5 MiB/s wr, 634 op/s 2024-04-14T13:38:09.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:08 smithi012 ceph-mon[27236]: pgmap v21: 97 pgs: 97 active+clean; 325 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.5 MiB/s wr, 634 op/s 2024-04-14T13:38:09.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 91671 13.82 MB/sec execute 43 sec latency 26.698 ms 2024-04-14T13:38:09.687 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:38:09.687 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 starting - - - - 2024-04-14T13:38:09.687 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (9m) 22s ago 9m 6405k - 17.2.0 e1d6a67b021e 68b810a9ff93 2024-04-14T13:38:09.687 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (8m) 24s ago 8m 6400k - 17.2.0 e1d6a67b021e 6bc31c51b938 2024-04-14T13:38:09.687 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (7m) 22s ago 8m 55.4M - 8.3.5 dad864ee21e9 c7b42d05cedc 2024-04-14T13:38:09.687 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (3m) 22s ago 3m 36.7M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:38:09.687 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (3m) 24s ago 3m 12.5M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:38:09.687 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:9283 running (10m) 22s ago 10m 315M - 17.2.0 e1d6a67b021e 074f98e540c3 2024-04-14T13:38:09.687 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283 running (47s) 24s ago 8m 491M - 19.0.0-2778-ga971fd5f b96c509eef84 089c1c1b8529 2024-04-14T13:38:09.688 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (10m) 22s ago 10m 53.3M 2048M 17.2.0 e1d6a67b021e 7786c27a60d0 2024-04-14T13:38:09.688 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (8m) 24s ago 8m 40.3M 2048M 17.2.0 e1d6a67b021e f6cd483dfa31 2024-04-14T13:38:09.688 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (2m) 22s ago 2m 252M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:38:09.688 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (2m) 24s ago 2m 76.0M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:38:09.688 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (8m) 22s ago 8m 17.4M - 1.3.1 1dbe0e931976 2338da898186 2024-04-14T13:38:09.688 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (8m) 24s ago 8m 17.8M - 1.3.1 1dbe0e931976 c2fae00595dd 2024-04-14T13:38:09.688 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (7m) 22s ago 7m 310M 1937M 17.2.0 e1d6a67b021e 3b8cdea7890c 2024-04-14T13:38:09.688 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (7m) 22s ago 7m 248M 1937M 17.2.0 e1d6a67b021e be86642c7790 2024-04-14T13:38:09.688 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (6m) 22s ago 6m 184M 1937M 17.2.0 e1d6a67b021e b1017f00578f 2024-04-14T13:38:09.688 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (6m) 22s ago 6m 263M 1937M 17.2.0 e1d6a67b021e 66fca4b191be 2024-04-14T13:38:09.688 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (5m) 24s ago 5m 298M 2705M 17.2.0 e1d6a67b021e 43b02cb64951 2024-04-14T13:38:09.688 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (5m) 24s ago 5m 315M 2705M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:38:09.688 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (5m) 24s ago 5m 267M 2705M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:38:09.689 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (4m) 24s ago 4m 388M 2705M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:38:09.689 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (7m) 22s ago 7m 64.6M - 2.33.4 514e6a882f6e 11fca64796ed 2024-04-14T13:38:10.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 92387 14.07 MB/sec execute 44 sec latency 21.678 ms 2024-04-14T13:38:10.374 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:10 smithi008 ceph-mon[24786]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:38:10.374 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:10 smithi008 ceph-mon[24786]: from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:38:10.374 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:10 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:38:10.374 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:10 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:38:10.374 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:10 smithi008 ceph-mon[24786]: Reconfiguring grafana.smithi008 (dependencies changed)... 2024-04-14T13:38:10.374 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:10 smithi008 ceph-mon[24786]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-14T13:38:10.374 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:10 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:38:10.375 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:10 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:38:10.375 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:10 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T13:38:10.375 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:10 smithi008 ceph-mon[24786]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T13:38:10.375 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:10 smithi008 ceph-mon[24786]: Reconfiguring daemon grafana.smithi008 on smithi008 2024-04-14T13:38:10.426 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:38:10.426 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:38:10.426 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:38:10.426 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:38:10.426 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:38:10.426 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-14T13:38:10.426 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T13:38:10.426 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:38:10.426 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:38:10.426 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T13:38:10.427 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:38:10.427 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:38:10.427 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:38:10.427 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:38:10.427 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:38:10.427 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-14T13:38:10.427 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T13:38:10.427 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:38:10.427 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:38:10.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:10 smithi012 ceph-mon[27236]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:38:10.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:10 smithi012 ceph-mon[27236]: from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:38:10.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:10 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:38:10.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:10 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:38:10.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:10 smithi012 ceph-mon[27236]: Reconfiguring grafana.smithi008 (dependencies changed)... 2024-04-14T13:38:10.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:10 smithi012 ceph-mon[27236]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-14T13:38:10.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:10 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:38:10.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:10 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:38:10.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:10 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T13:38:10.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:10 smithi012 ceph-mon[27236]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T13:38:10.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:10 smithi012 ceph-mon[27236]: Reconfiguring daemon grafana.smithi008 on smithi008 2024-04-14T13:38:10.959 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:38:10.960 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:38:10.960 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:38:10.960 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:38:10.960 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-04-14T13:38:10.960 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "1/23 daemons upgraded", 2024-04-14T13:38:10.960 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "", 2024-04-14T13:38:10.960 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:38:10.960 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:38:11.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 92735 14.02 MB/sec execute 45 sec latency 50.011 ms 2024-04-14T13:38:11.382 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:11 smithi008 ceph-mon[24786]: from='client.24517 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:38:11.382 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:11 smithi008 ceph-mon[24786]: pgmap v22: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.5 MiB/s wr, 948 op/s 2024-04-14T13:38:11.382 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:11 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/211055270' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:38:11.496 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-04-14T13:38:11.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:11 smithi012 ceph-mon[27236]: from='client.24517 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:38:11.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:11 smithi012 ceph-mon[27236]: pgmap v22: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.5 MiB/s wr, 948 op/s 2024-04-14T13:38:11.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:11 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/211055270' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:38:12.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 93171 13.84 MB/sec execute 46 sec latency 39.459 ms 2024-04-14T13:38:12.390 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:12 smithi008 ceph-mon[24786]: from='client.24525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:38:12.390 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:12 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/4047068528' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:38:12.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:12 smithi012 ceph-mon[27236]: from='client.24525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:38:12.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:12 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/4047068528' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:38:13.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 93659 13.74 MB/sec execute 47 sec latency 39.912 ms 2024-04-14T13:38:13.397 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:13 smithi008 ceph-mon[24786]: pgmap v23: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 5.9 MiB/s wr, 623 op/s 2024-04-14T13:38:13.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:13 smithi012 ceph-mon[27236]: pgmap v23: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 5.9 MiB/s wr, 623 op/s 2024-04-14T13:38:14.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 94157 13.74 MB/sec execute 48 sec latency 57.574 ms 2024-04-14T13:38:15.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 94748 13.69 MB/sec execute 49 sec latency 40.934 ms 2024-04-14T13:38:15.426 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:15 smithi008 ceph-mon[24786]: pgmap v24: 97 pgs: 97 active+clean; 334 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 932 op/s 2024-04-14T13:38:15.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:15 smithi012 ceph-mon[27236]: pgmap v24: 97 pgs: 97 active+clean; 334 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 932 op/s 2024-04-14T13:38:16.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 95318 13.83 MB/sec execute 50 sec latency 46.042 ms 2024-04-14T13:38:16.685 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:16 smithi008 ceph-mon[24786]: pgmap v25: 97 pgs: 97 active+clean; 334 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.5 MiB/s wr, 622 op/s 2024-04-14T13:38:17.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:16 smithi012 ceph-mon[27236]: pgmap v25: 97 pgs: 97 active+clean; 334 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.5 MiB/s wr, 622 op/s 2024-04-14T13:38:17.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 95938 14.02 MB/sec execute 51 sec latency 24.260 ms 2024-04-14T13:38:17.696 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:17 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:38:18.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:17 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:38:18.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 96286 13.98 MB/sec execute 52 sec latency 48.110 ms 2024-04-14T13:38:18.701 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:18 smithi008 ceph-mon[24786]: pgmap v26: 97 pgs: 97 active+clean; 334 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.5 MiB/s wr, 622 op/s 2024-04-14T13:38:19.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:18 smithi012 ceph-mon[27236]: pgmap v26: 97 pgs: 97 active+clean; 334 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.5 MiB/s wr, 622 op/s 2024-04-14T13:38:19.309 INFO:teuthology.orchestra.run.smithi008.stdout: 5 96621 13.81 MB/sec execute 53 sec latency 87.961 ms 2024-04-14T13:38:20.310 INFO:teuthology.orchestra.run.smithi008.stdout: 5 96953 13.62 MB/sec execute 54 sec latency 49.796 ms 2024-04-14T13:38:20.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:20 smithi008 ceph-mon[24786]: pgmap v27: 97 pgs: 97 active+clean; 334 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.9 MiB/s wr, 943 op/s 2024-04-14T13:38:21.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:20 smithi012 ceph-mon[27236]: pgmap v27: 97 pgs: 97 active+clean; 334 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.9 MiB/s wr, 943 op/s 2024-04-14T13:38:21.310 INFO:teuthology.orchestra.run.smithi008.stdout: 5 97343 13.58 MB/sec execute 55 sec latency 37.275 ms 2024-04-14T13:38:22.310 INFO:teuthology.orchestra.run.smithi008.stdout: 5 97887 13.58 MB/sec execute 56 sec latency 37.872 ms 2024-04-14T13:38:23.004 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:22 smithi008 ceph-mon[24786]: pgmap v28: 97 pgs: 97 active+clean; 334 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.9 MiB/s wr, 629 op/s 2024-04-14T13:38:23.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:22 smithi012 ceph-mon[27236]: pgmap v28: 97 pgs: 97 active+clean; 334 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.9 MiB/s wr, 629 op/s 2024-04-14T13:38:23.310 INFO:teuthology.orchestra.run.smithi008.stdout: 5 98390 13.46 MB/sec execute 57 sec latency 45.625 ms 2024-04-14T13:38:24.310 INFO:teuthology.orchestra.run.smithi008.stdout: 5 98855 13.57 MB/sec execute 58 sec latency 30.802 ms 2024-04-14T13:38:25.041 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:24 smithi008 ceph-mon[24786]: pgmap v29: 97 pgs: 97 active+clean; 320 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 880 op/s 2024-04-14T13:38:25.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:24 smithi012 ceph-mon[27236]: pgmap v29: 97 pgs: 97 active+clean; 320 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 880 op/s 2024-04-14T13:38:25.314 INFO:teuthology.orchestra.run.smithi008.stdout: 5 99375 13.69 MB/sec execute 59 sec latency 37.649 ms 2024-04-14T13:38:26.314 INFO:teuthology.orchestra.run.smithi008.stdout: 5 99712 13.65 MB/sec execute 60 sec latency 46.242 ms 2024-04-14T13:38:26.823 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:26 smithi008 ceph-mon[24786]: pgmap v30: 97 pgs: 97 active+clean; 320 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.5 MiB/s wr, 572 op/s 2024-04-14T13:38:27.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:26 smithi012 ceph-mon[27236]: pgmap v30: 97 pgs: 97 active+clean; 320 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.5 MiB/s wr, 572 op/s 2024-04-14T13:38:27.315 INFO:teuthology.orchestra.run.smithi008.stdout: 5 99943 13.49 MB/sec execute 61 sec latency 47.635 ms 2024-04-14T13:38:28.315 INFO:teuthology.orchestra.run.smithi008.stdout: 5 100289 13.37 MB/sec execute 62 sec latency 48.368 ms 2024-04-14T13:38:29.255 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:29 smithi008 ceph-mon[24786]: pgmap v31: 97 pgs: 97 active+clean; 320 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.5 MiB/s wr, 572 op/s 2024-04-14T13:38:29.256 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:29 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:38:29.256 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:29 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:38:29.256 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:29 smithi008 ceph-mon[24786]: Reconfiguring node-exporter.smithi008 (dependencies changed)... 2024-04-14T13:38:29.256 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:29 smithi008 ceph-mon[24786]: Reconfiguring daemon node-exporter.smithi008 on smithi008 2024-04-14T13:38:29.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:29 smithi012 ceph-mon[27236]: pgmap v31: 97 pgs: 97 active+clean; 320 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.5 MiB/s wr, 572 op/s 2024-04-14T13:38:29.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:29 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:38:29.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:29 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:38:29.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:29 smithi012 ceph-mon[27236]: Reconfiguring node-exporter.smithi008 (dependencies changed)... 2024-04-14T13:38:29.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:29 smithi012 ceph-mon[27236]: Reconfiguring daemon node-exporter.smithi008 on smithi008 2024-04-14T13:38:29.314 INFO:teuthology.orchestra.run.smithi008.stdout: 5 100747 13.29 MB/sec execute 63 sec latency 50.257 ms 2024-04-14T13:38:30.315 INFO:teuthology.orchestra.run.smithi008.stdout: 5 101139 13.20 MB/sec execute 64 sec latency 62.692 ms 2024-04-14T13:38:31.046 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:31 smithi008 ceph-mon[24786]: pgmap v32: 97 pgs: 97 active+clean; 332 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 811 op/s 2024-04-14T13:38:31.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:31 smithi012 ceph-mon[27236]: pgmap v32: 97 pgs: 97 active+clean; 332 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 811 op/s 2024-04-14T13:38:31.315 INFO:teuthology.orchestra.run.smithi008.stdout: 5 101691 13.22 MB/sec execute 65 sec latency 44.656 ms 2024-04-14T13:38:32.309 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:38:32.315 INFO:teuthology.orchestra.run.smithi008.stdout: 5 102130 13.23 MB/sec execute 66 sec latency 30.881 ms 2024-04-14T13:38:32.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:38:33.070 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:33 smithi008 ceph-mon[24786]: pgmap v33: 97 pgs: 97 active+clean; 332 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.0 MiB/s wr, 489 op/s 2024-04-14T13:38:33.315 INFO:teuthology.orchestra.run.smithi008.stdout: 5 102571 13.29 MB/sec execute 67 sec latency 43.526 ms 2024-04-14T13:38:33.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:33 smithi012 ceph-mon[27236]: pgmap v33: 97 pgs: 97 active+clean; 332 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.0 MiB/s wr, 489 op/s 2024-04-14T13:38:34.315 INFO:teuthology.orchestra.run.smithi008.stdout: 5 102893 13.28 MB/sec execute 68 sec latency 144.650 ms 2024-04-14T13:38:35.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:35 smithi008 ceph-mon[24786]: pgmap v34: 97 pgs: 97 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 754 op/s 2024-04-14T13:38:35.315 INFO:teuthology.orchestra.run.smithi008.stdout: 5 103288 13.26 MB/sec execute 69 sec latency 35.921 ms 2024-04-14T13:38:35.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:35 smithi012 ceph-mon[27236]: pgmap v34: 97 pgs: 97 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 754 op/s 2024-04-14T13:38:36.315 INFO:teuthology.orchestra.run.smithi008.stdout: 5 103643 13.18 MB/sec execute 70 sec latency 34.046 ms 2024-04-14T13:38:37.194 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:36 smithi008 ceph-mon[24786]: pgmap v35: 97 pgs: 97 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.3 MiB/s wr, 503 op/s 2024-04-14T13:38:37.194 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:36 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:38:37.194 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:36 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:38:37.194 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:36 smithi008 ceph-mon[24786]: Reconfiguring prometheus.smithi008 (dependencies changed)... 2024-04-14T13:38:37.194 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:36 smithi008 ceph-mon[24786]: Reconfiguring daemon prometheus.smithi008 on smithi008 2024-04-14T13:38:37.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:36 smithi012 ceph-mon[27236]: pgmap v35: 97 pgs: 97 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.3 MiB/s wr, 503 op/s 2024-04-14T13:38:37.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:36 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:38:37.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:36 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:38:37.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:36 smithi012 ceph-mon[27236]: Reconfiguring prometheus.smithi008 (dependencies changed)... 2024-04-14T13:38:37.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:36 smithi012 ceph-mon[27236]: Reconfiguring daemon prometheus.smithi008 on smithi008 2024-04-14T13:38:37.315 INFO:teuthology.orchestra.run.smithi008.stdout: 5 103975 13.04 MB/sec execute 71 sec latency 39.657 ms 2024-04-14T13:38:38.315 INFO:teuthology.orchestra.run.smithi008.stdout: 5 104361 12.96 MB/sec execute 72 sec latency 33.664 ms 2024-04-14T13:38:38.971 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:38 smithi008 ceph-mon[24786]: pgmap v36: 97 pgs: 97 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.3 MiB/s wr, 503 op/s 2024-04-14T13:38:39.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:38 smithi012 ceph-mon[27236]: pgmap v36: 97 pgs: 97 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.3 MiB/s wr, 503 op/s 2024-04-14T13:38:39.315 INFO:teuthology.orchestra.run.smithi008.stdout: 5 104745 12.92 MB/sec execute 73 sec latency 42.601 ms 2024-04-14T13:38:40.317 INFO:teuthology.orchestra.run.smithi008.stdout: 5 105172 12.89 MB/sec execute 74 sec latency 56.379 ms 2024-04-14T13:38:41.022 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:40 smithi008 ceph-mon[24786]: pgmap v37: 97 pgs: 97 active+clean; 330 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 731 op/s 2024-04-14T13:38:41.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:40 smithi012 ceph-mon[27236]: pgmap v37: 97 pgs: 97 active+clean; 330 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 731 op/s 2024-04-14T13:38:41.317 INFO:teuthology.orchestra.run.smithi008.stdout: 5 105639 12.90 MB/sec execute 75 sec latency 31.680 ms 2024-04-14T13:38:42.071 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:38:42.317 INFO:teuthology.orchestra.run.smithi008.stdout: 5 106168 12.98 MB/sec execute 76 sec latency 23.358 ms 2024-04-14T13:38:43.044 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:42 smithi008 ceph-mon[24786]: pgmap v38: 97 pgs: 97 active+clean; 330 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.9 MiB/s wr, 492 op/s 2024-04-14T13:38:43.044 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:42 smithi008 ceph-mon[24786]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:38:43.151 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:38:43.152 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 starting - - - - 2024-04-14T13:38:43.152 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (9m) 56s ago 9m 6405k - 17.2.0 e1d6a67b021e 68b810a9ff93 2024-04-14T13:38:43.152 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (8m) 57s ago 8m 6400k - 17.2.0 e1d6a67b021e 6bc31c51b938 2024-04-14T13:38:43.152 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 starting - - - - 2024-04-14T13:38:43.152 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (3m) 56s ago 3m 36.7M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:38:43.152 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (3m) 57s ago 3m 12.5M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:38:43.152 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:9283 running (10m) 56s ago 10m 315M - 17.2.0 e1d6a67b021e 074f98e540c3 2024-04-14T13:38:43.152 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283 running (80s) 57s ago 8m 491M - 19.0.0-2778-ga971fd5f b96c509eef84 089c1c1b8529 2024-04-14T13:38:43.152 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (11m) 56s ago 11m 53.3M 2048M 17.2.0 e1d6a67b021e 7786c27a60d0 2024-04-14T13:38:43.152 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (8m) 57s ago 8m 40.3M 2048M 17.2.0 e1d6a67b021e f6cd483dfa31 2024-04-14T13:38:43.152 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (3m) 56s ago 3m 252M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:38:43.152 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (3m) 57s ago 3m 76.0M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:38:43.152 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 starting - - - - 2024-04-14T13:38:43.152 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (8m) 57s ago 8m 17.8M - 1.3.1 1dbe0e931976 c2fae00595dd 2024-04-14T13:38:43.153 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (8m) 56s ago 8m 310M 1937M 17.2.0 e1d6a67b021e 3b8cdea7890c 2024-04-14T13:38:43.153 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (7m) 56s ago 7m 248M 1937M 17.2.0 e1d6a67b021e be86642c7790 2024-04-14T13:38:43.153 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (7m) 56s ago 7m 184M 1937M 17.2.0 e1d6a67b021e b1017f00578f 2024-04-14T13:38:43.153 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (6m) 56s ago 6m 263M 1937M 17.2.0 e1d6a67b021e 66fca4b191be 2024-04-14T13:38:43.153 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (6m) 57s ago 6m 298M 2705M 17.2.0 e1d6a67b021e 43b02cb64951 2024-04-14T13:38:43.153 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (6m) 57s ago 6m 315M 2705M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:38:43.153 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (5m) 57s ago 5m 267M 2705M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:38:43.153 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (5m) 57s ago 5m 388M 2705M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:38:43.153 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (8m) 56s ago 8m 64.6M - 2.33.4 514e6a882f6e 11fca64796ed 2024-04-14T13:38:43.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:42 smithi012 ceph-mon[27236]: pgmap v38: 97 pgs: 97 active+clean; 330 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.9 MiB/s wr, 492 op/s 2024-04-14T13:38:43.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:42 smithi012 ceph-mon[27236]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:38:43.317 INFO:teuthology.orchestra.run.smithi008.stdout: 5 106622 13.01 MB/sec execute 77 sec latency 38.158 ms 2024-04-14T13:38:43.893 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:38:43.893 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:38:43.893 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:38:43.893 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:38:43.893 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:38:43.893 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-14T13:38:43.893 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T13:38:43.893 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:38:43.893 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:38:43.893 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T13:38:43.893 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:38:43.893 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:38:43.893 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:38:43.894 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:38:43.894 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:38:43.894 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-14T13:38:43.894 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T13:38:43.894 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:38:43.894 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:38:44.054 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:43 smithi008 ceph-mon[24786]: from='client.24535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:38:44.054 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:43 smithi008 ceph-mon[24786]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:38:44.055 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:43 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2280139472' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:38:44.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:43 smithi012 ceph-mon[27236]: from='client.24535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:38:44.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:43 smithi012 ceph-mon[27236]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:38:44.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:43 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/2280139472' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:38:44.317 INFO:teuthology.orchestra.run.smithi008.stdout: 5 107026 13.00 MB/sec execute 78 sec latency 29.073 ms 2024-04-14T13:38:44.498 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:38:44.498 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:38:44.498 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:38:44.498 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:38:44.498 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-04-14T13:38:44.498 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "1/23 daemons upgraded", 2024-04-14T13:38:44.498 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "", 2024-04-14T13:38:44.498 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:38:44.499 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:38:45.058 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:44 smithi008 ceph-mon[24786]: pgmap v39: 97 pgs: 97 active+clean; 334 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 750 op/s 2024-04-14T13:38:45.058 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:44 smithi008 ceph-mon[24786]: from='client.24541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:38:45.225 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-04-14T13:38:45.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:44 smithi012 ceph-mon[27236]: pgmap v39: 97 pgs: 97 active+clean; 334 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 750 op/s 2024-04-14T13:38:45.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:44 smithi012 ceph-mon[27236]: from='client.24541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:38:45.317 INFO:teuthology.orchestra.run.smithi008.stdout: 5 107462 12.92 MB/sec execute 79 sec latency 43.463 ms 2024-04-14T13:38:46.078 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:45 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/3572531621' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:38:46.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:45 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/3572531621' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:38:46.317 INFO:teuthology.orchestra.run.smithi008.stdout: 5 107882 12.86 MB/sec execute 80 sec latency 64.837 ms 2024-04-14T13:38:47.086 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:46 smithi008 ceph-mon[24786]: pgmap v40: 97 pgs: 97 active+clean; 334 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.2 MiB/s wr, 486 op/s 2024-04-14T13:38:47.086 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:46 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:38:47.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:46 smithi012 ceph-mon[27236]: pgmap v40: 97 pgs: 97 active+clean; 334 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.2 MiB/s wr, 486 op/s 2024-04-14T13:38:47.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:46 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:38:47.317 INFO:teuthology.orchestra.run.smithi008.stdout: 5 108452 12.92 MB/sec execute 81 sec latency 32.326 ms 2024-04-14T13:38:48.317 INFO:teuthology.orchestra.run.smithi008.stdout: 5 109033 12.89 MB/sec execute 82 sec latency 31.530 ms 2024-04-14T13:38:49.121 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:48 smithi008 ceph-mon[24786]: pgmap v41: 97 pgs: 97 active+clean; 334 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.2 MiB/s wr, 486 op/s 2024-04-14T13:38:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:48 smithi012 ceph-mon[27236]: pgmap v41: 97 pgs: 97 active+clean; 334 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.2 MiB/s wr, 486 op/s 2024-04-14T13:38:49.317 INFO:teuthology.orchestra.run.smithi008.stdout: 5 109412 12.90 MB/sec execute 83 sec latency 38.242 ms 2024-04-14T13:38:50.318 INFO:teuthology.orchestra.run.smithi008.stdout: 5 109796 12.95 MB/sec execute 84 sec latency 39.376 ms 2024-04-14T13:38:51.173 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:50 smithi008 ceph-mon[24786]: pgmap v42: 97 pgs: 97 active+clean; 327 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 778 op/s 2024-04-14T13:38:51.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:50 smithi012 ceph-mon[27236]: pgmap v42: 97 pgs: 97 active+clean; 327 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 778 op/s 2024-04-14T13:38:51.317 INFO:teuthology.orchestra.run.smithi008.stdout: 5 110141 12.91 MB/sec execute 85 sec latency 41.152 ms 2024-04-14T13:38:52.318 INFO:teuthology.orchestra.run.smithi008.stdout: 5 110498 12.90 MB/sec execute 86 sec latency 40.683 ms 2024-04-14T13:38:53.207 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:52 smithi008 ceph-mon[24786]: pgmap v43: 97 pgs: 97 active+clean; 327 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.4 MiB/s wr, 550 op/s 2024-04-14T13:38:53.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:52 smithi012 ceph-mon[27236]: pgmap v43: 97 pgs: 97 active+clean; 327 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.4 MiB/s wr, 550 op/s 2024-04-14T13:38:53.317 INFO:teuthology.orchestra.run.smithi008.stdout: 5 110833 12.81 MB/sec execute 87 sec latency 39.572 ms 2024-04-14T13:38:54.317 INFO:teuthology.orchestra.run.smithi008.stdout: 5 111290 12.76 MB/sec execute 88 sec latency 47.967 ms 2024-04-14T13:38:55.062 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:54 smithi008 ceph-mon[24786]: pgmap v44: 97 pgs: 97 active+clean; 330 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 782 op/s 2024-04-14T13:38:55.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:54 smithi012 ceph-mon[27236]: pgmap v44: 97 pgs: 97 active+clean; 330 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 782 op/s 2024-04-14T13:38:55.324 INFO:teuthology.orchestra.run.smithi008.stdout: 5 111713 12.74 MB/sec execute 89 sec latency 36.761 ms 2024-04-14T13:38:56.324 INFO:teuthology.orchestra.run.smithi008.stdout: 5 112151 12.71 MB/sec execute 90 sec latency 85.903 ms 2024-04-14T13:38:56.820 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:56 smithi008 ceph-mon[24786]: pgmap v45: 97 pgs: 97 active+clean; 330 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.4 MiB/s wr, 524 op/s 2024-04-14T13:38:57.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:56 smithi012 ceph-mon[27236]: pgmap v45: 97 pgs: 97 active+clean; 330 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.4 MiB/s wr, 524 op/s 2024-04-14T13:38:57.324 INFO:teuthology.orchestra.run.smithi008.stdout: 5 112569 12.66 MB/sec execute 91 sec latency 42.225 ms 2024-04-14T13:38:58.324 INFO:teuthology.orchestra.run.smithi008.stdout: 5 112984 12.69 MB/sec execute 92 sec latency 44.773 ms 2024-04-14T13:38:58.850 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:38:58 smithi008 ceph-mon[24786]: pgmap v46: 97 pgs: 97 active+clean; 330 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.4 MiB/s wr, 524 op/s 2024-04-14T13:38:59.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:38:58 smithi012 ceph-mon[27236]: pgmap v46: 97 pgs: 97 active+clean; 330 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.4 MiB/s wr, 524 op/s 2024-04-14T13:38:59.324 INFO:teuthology.orchestra.run.smithi008.stdout: 5 113441 12.74 MB/sec execute 93 sec latency 35.878 ms 2024-04-14T13:39:00.324 INFO:teuthology.orchestra.run.smithi008.stdout: 5 113955 12.80 MB/sec execute 94 sec latency 29.521 ms 2024-04-14T13:39:00.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:00 smithi008 ceph-mon[24786]: pgmap v47: 97 pgs: 97 active+clean; 326 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.8 MiB/s wr, 783 op/s 2024-04-14T13:39:01.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:00 smithi012 ceph-mon[27236]: pgmap v47: 97 pgs: 97 active+clean; 326 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.8 MiB/s wr, 783 op/s 2024-04-14T13:39:01.324 INFO:teuthology.orchestra.run.smithi008.stdout: 5 114386 12.74 MB/sec execute 95 sec latency 32.075 ms 2024-04-14T13:39:02.324 INFO:teuthology.orchestra.run.smithi008.stdout: 5 115021 12.74 MB/sec execute 96 sec latency 57.666 ms 2024-04-14T13:39:02.895 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:02 smithi008 ceph-mon[24786]: pgmap v48: 97 pgs: 97 active+clean; 326 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.1 MiB/s wr, 491 op/s 2024-04-14T13:39:02.895 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:02 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:39:02.895 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:02 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:02.895 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:02 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:02.895 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:02 smithi008 ceph-mon[24786]: Reconfiguring node-exporter.smithi012 (dependencies changed)... 2024-04-14T13:39:02.895 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:02 smithi008 ceph-mon[24786]: Reconfiguring daemon node-exporter.smithi012 on smithi012 2024-04-14T13:39:03.042 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:02 smithi012 ceph-mon[27236]: pgmap v48: 97 pgs: 97 active+clean; 326 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.1 MiB/s wr, 491 op/s 2024-04-14T13:39:03.042 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:02 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:39:03.042 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:02 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:03.042 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:02 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:03.042 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:02 smithi012 ceph-mon[27236]: Reconfiguring node-exporter.smithi012 (dependencies changed)... 2024-04-14T13:39:03.042 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:02 smithi012 ceph-mon[27236]: Reconfiguring daemon node-exporter.smithi012 on smithi012 2024-04-14T13:39:03.324 INFO:teuthology.orchestra.run.smithi008.stdout: 5 115871 12.81 MB/sec execute 97 sec latency 16.061 ms 2024-04-14T13:39:04.324 INFO:teuthology.orchestra.run.smithi008.stdout: 5 116531 12.89 MB/sec execute 98 sec latency 30.607 ms 2024-04-14T13:39:04.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: pgmap v49: 97 pgs: 97 active+clean; 335 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 813 op/s 2024-04-14T13:39:04.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: mgrmap e27: smithi012.ksvnxm(active, since 92s), standbys: smithi008.yemlhr 2024-04-14T13:39:04.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:04.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:04.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T13:39:04.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T13:39:04.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T13:39:04.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T13:39:04.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:04.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T13:39:04.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T13:39:04.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi008.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T13:39:04.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi008.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T13:39:04.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:04.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:39:04.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:39:04.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi008.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T13:39:04.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi008.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T13:39:04.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:04.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:39:04.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: Upgrade: Need to upgrade myself (mgr.smithi012.ksvnxm) 2024-04-14T13:39:04.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:04 smithi008 ceph-mon[24786]: Upgrade: Need to upgrade myself (mgr.smithi012.ksvnxm) 2024-04-14T13:39:05.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: pgmap v49: 97 pgs: 97 active+clean; 335 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 813 op/s 2024-04-14T13:39:05.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: mgrmap e27: smithi012.ksvnxm(active, since 92s), standbys: smithi008.yemlhr 2024-04-14T13:39:05.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T13:39:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T13:39:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T13:39:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T13:39:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T13:39:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T13:39:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi008.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T13:39:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi008.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T13:39:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:39:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:39:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi008.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T13:39:05.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi008.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T13:39:05.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:05.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:39:05.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: Upgrade: Need to upgrade myself (mgr.smithi012.ksvnxm) 2024-04-14T13:39:05.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:04 smithi012 ceph-mon[27236]: Upgrade: Need to upgrade myself (mgr.smithi012.ksvnxm) 2024-04-14T13:39:05.324 INFO:teuthology.orchestra.run.smithi008.stdout: 5 117087 12.98 MB/sec execute 99 sec latency 39.139 ms 2024-04-14T13:39:06.325 INFO:teuthology.orchestra.run.smithi008.stdout: 5 117539 12.99 MB/sec execute 100 sec latency 32.258 ms 2024-04-14T13:39:06.677 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:06 smithi008 ceph-mon[24786]: pgmap v50: 97 pgs: 97 active+clean; 335 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.7 MiB/s wr, 580 op/s 2024-04-14T13:39:07.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:06 smithi012 ceph-mon[27236]: pgmap v50: 97 pgs: 97 active+clean; 335 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.7 MiB/s wr, 580 op/s 2024-04-14T13:39:07.324 INFO:teuthology.orchestra.run.smithi008.stdout: 5 117870 12.92 MB/sec execute 101 sec latency 55.322 ms 2024-04-14T13:39:08.332 INFO:teuthology.orchestra.run.smithi008.stdout: 5 118495 12.90 MB/sec execute 102 sec latency 25.202 ms 2024-04-14T13:39:08.437 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:08 smithi008 ceph-mon[24786]: Upgrade: Updating mgr.smithi008.yemlhr 2024-04-14T13:39:08.437 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:08 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:08.438 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:08 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.yemlhr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:39:08.438 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:08 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.yemlhr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:39:08.438 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:08 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:39:08.438 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:08 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:39:08.438 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:08 smithi008 ceph-mon[24786]: Deploying daemon mgr.smithi008.yemlhr on smithi008 2024-04-14T13:39:08.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:08 smithi012 ceph-mon[27236]: Upgrade: Updating mgr.smithi008.yemlhr 2024-04-14T13:39:08.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:08 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:08.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:08 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.yemlhr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:39:08.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:08 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.yemlhr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:39:08.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:08 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:39:08.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:08 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:39:08.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:08 smithi012 ceph-mon[27236]: Deploying daemon mgr.smithi008.yemlhr on smithi008 2024-04-14T13:39:09.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 119439 13.02 MB/sec execute 103 sec latency 26.909 ms 2024-04-14T13:39:09.442 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:09 smithi008 ceph-mon[24786]: pgmap v51: 97 pgs: 97 active+clean; 335 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.7 MiB/s wr, 580 op/s 2024-04-14T13:39:09.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:09 smithi012 ceph-mon[27236]: pgmap v51: 97 pgs: 97 active+clean; 335 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.7 MiB/s wr, 580 op/s 2024-04-14T13:39:10.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 120071 13.06 MB/sec execute 104 sec latency 30.558 ms 2024-04-14T13:39:11.318 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:11 smithi008 ceph-mon[24786]: pgmap v52: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 893 op/s 2024-04-14T13:39:11.332 INFO:teuthology.orchestra.run.smithi008.stdout: 5 120663 13.15 MB/sec execute 105 sec latency 26.800 ms 2024-04-14T13:39:11.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:11 smithi012 ceph-mon[27236]: pgmap v52: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 893 op/s 2024-04-14T13:39:12.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 121090 13.18 MB/sec execute 106 sec latency 37.284 ms 2024-04-14T13:39:13.267 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:13 smithi008 ceph-mon[24786]: pgmap v53: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.7 MiB/s wr, 634 op/s 2024-04-14T13:39:13.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 121437 13.09 MB/sec execute 107 sec latency 64.594 ms 2024-04-14T13:39:13.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:13 smithi012 ceph-mon[27236]: pgmap v53: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.7 MiB/s wr, 634 op/s 2024-04-14T13:39:14.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 122075 13.09 MB/sec execute 108 sec latency 37.278 ms 2024-04-14T13:39:15.278 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:15 smithi008 ceph-mon[24786]: pgmap v54: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 995 op/s 2024-04-14T13:39:15.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 122668 13.10 MB/sec execute 109 sec latency 97.848 ms 2024-04-14T13:39:15.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:15 smithi012 ceph-mon[27236]: pgmap v54: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 995 op/s 2024-04-14T13:39:15.813 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:39:16.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 123076 13.07 MB/sec execute 110 sec latency 48.168 ms 2024-04-14T13:39:16.781 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:16 smithi008 ceph-mon[24786]: from='client.24547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:39:16.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:16 smithi008 ceph-mon[24786]: pgmap v55: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 7.3 MiB/s wr, 673 op/s 2024-04-14T13:39:16.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:16 smithi008 ceph-mon[24786]: from='client.24551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:39:16.875 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:39:16.875 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 starting - - - - 2024-04-14T13:39:16.875 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (10m) 89s ago 10m 6405k - 17.2.0 e1d6a67b021e 68b810a9ff93 2024-04-14T13:39:16.875 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (9m) 91s ago 9m 6400k - 17.2.0 e1d6a67b021e 6bc31c51b938 2024-04-14T13:39:16.875 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 starting - - - - 2024-04-14T13:39:16.875 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (4m) 89s ago 4m 36.7M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:39:16.876 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (4m) 91s ago 4m 12.5M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:39:16.876 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:9283 running (11m) 89s ago 11m 315M - 17.2.0 e1d6a67b021e 074f98e540c3 2024-04-14T13:39:16.876 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283 running (114s) 91s ago 9m 491M - 19.0.0-2778-ga971fd5f b96c509eef84 089c1c1b8529 2024-04-14T13:39:16.876 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (11m) 89s ago 11m 53.3M 2048M 17.2.0 e1d6a67b021e 7786c27a60d0 2024-04-14T13:39:16.876 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (9m) 91s ago 9m 40.3M 2048M 17.2.0 e1d6a67b021e f6cd483dfa31 2024-04-14T13:39:16.876 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (4m) 89s ago 4m 252M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:39:16.876 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (3m) 91s ago 3m 76.0M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:39:16.876 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 starting - - - - 2024-04-14T13:39:16.876 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 starting - - - - 2024-04-14T13:39:16.876 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (8m) 89s ago 8m 310M 1937M 17.2.0 e1d6a67b021e 3b8cdea7890c 2024-04-14T13:39:16.876 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (8m) 89s ago 8m 248M 1937M 17.2.0 e1d6a67b021e be86642c7790 2024-04-14T13:39:16.876 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (7m) 89s ago 7m 184M 1937M 17.2.0 e1d6a67b021e b1017f00578f 2024-04-14T13:39:16.876 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (7m) 89s ago 7m 263M 1937M 17.2.0 e1d6a67b021e 66fca4b191be 2024-04-14T13:39:16.876 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (6m) 91s ago 6m 298M 2705M 17.2.0 e1d6a67b021e 43b02cb64951 2024-04-14T13:39:16.877 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (6m) 91s ago 6m 315M 2705M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:39:16.877 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (6m) 91s ago 6m 267M 2705M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:39:16.877 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (5m) 91s ago 5m 388M 2705M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:39:16.877 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 starting - - - - 2024-04-14T13:39:17.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:16 smithi012 ceph-mon[27236]: from='client.24547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:39:17.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:16 smithi012 ceph-mon[27236]: pgmap v55: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 7.3 MiB/s wr, 673 op/s 2024-04-14T13:39:17.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:16 smithi012 ceph-mon[27236]: from='client.24551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:39:17.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 123510 13.08 MB/sec execute 111 sec latency 56.413 ms 2024-04-14T13:39:17.593 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:39:17.593 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:39:17.593 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:39:17.593 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:39:17.593 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:39:17.593 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-14T13:39:17.593 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T13:39:17.593 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:39:17.593 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:39:17.593 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T13:39:17.594 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:39:17.594 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:39:17.594 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:39:17.594 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:39:17.594 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:39:17.594 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-14T13:39:17.594 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T13:39:17.594 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:39:17.594 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:39:17.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:17 smithi012 ceph-mon[27236]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:39:17.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:17 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:39:17.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:17 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:17.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:17 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:17.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:17 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:39:17.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:17 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/2697851283' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:39:17.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:17 smithi008 ceph-mon[24786]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:39:17.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:17 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:39:17.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:17 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:17.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:17 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:17.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:17 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:39:17.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:17 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2697851283' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:39:18.077 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:39:18.077 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:39:18.077 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:39:18.077 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:39:18.078 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-04-14T13:39:18.078 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "", 2024-04-14T13:39:18.078 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading mgr daemons", 2024-04-14T13:39:18.078 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:39:18.078 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:39:18.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 123798 13.02 MB/sec execute 112 sec latency 52.212 ms 2024-04-14T13:39:18.781 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-04-14T13:39:18.826 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:18 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:18.826 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:18 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:18.826 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:18 smithi008 ceph-mon[24786]: pgmap v56: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 7.3 MiB/s wr, 673 op/s 2024-04-14T13:39:18.826 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:18 smithi008 ceph-mon[24786]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:39:19.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:18 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:19.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:18 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:19.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:18 smithi012 ceph-mon[27236]: pgmap v56: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 7.3 MiB/s wr, 673 op/s 2024-04-14T13:39:19.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:18 smithi012 ceph-mon[27236]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:39:19.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 124408 13.11 MB/sec execute 113 sec latency 55.277 ms 2024-04-14T13:39:19.838 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:19 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/1583027003' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:39:20.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:19 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/1583027003' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:39:20.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 124925 13.12 MB/sec execute 114 sec latency 37.241 ms 2024-04-14T13:39:20.850 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:20 smithi008 ceph-mon[24786]: pgmap v57: 97 pgs: 97 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 962 op/s 2024-04-14T13:39:21.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:20 smithi012 ceph-mon[27236]: pgmap v57: 97 pgs: 97 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 962 op/s 2024-04-14T13:39:21.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 125219 13.04 MB/sec execute 115 sec latency 41.051 ms 2024-04-14T13:39:22.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 125483 12.96 MB/sec execute 116 sec latency 54.884 ms 2024-04-14T13:39:22.901 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:22 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:22.902 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:22 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:22.902 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:22 smithi008 ceph-mon[24786]: pgmap v58: 97 pgs: 97 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.6 MiB/s wr, 649 op/s 2024-04-14T13:39:23.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:22 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:23.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:22 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:23.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:22 smithi012 ceph-mon[27236]: pgmap v58: 97 pgs: 97 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.6 MiB/s wr, 649 op/s 2024-04-14T13:39:23.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 125809 12.95 MB/sec execute 117 sec latency 57.422 ms 2024-04-14T13:39:24.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 126213 12.89 MB/sec execute 118 sec latency 47.166 ms 2024-04-14T13:39:24.951 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:24 smithi008 ceph-mon[24786]: pgmap v59: 97 pgs: 97 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 9.2 MiB/s wr, 893 op/s 2024-04-14T13:39:25.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:24 smithi012 ceph-mon[27236]: pgmap v59: 97 pgs: 97 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 9.2 MiB/s wr, 893 op/s 2024-04-14T13:39:25.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 126516 12.88 MB/sec execute 119 sec latency 52.391 ms 2024-04-14T13:39:26.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 126779 12.80 MB/sec execute 120 sec latency 69.415 ms 2024-04-14T13:39:26.735 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:26 smithi008 ceph-mon[24786]: pgmap v60: 97 pgs: 97 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.3 MiB/s wr, 532 op/s 2024-04-14T13:39:27.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:26 smithi012 ceph-mon[27236]: pgmap v60: 97 pgs: 97 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.3 MiB/s wr, 532 op/s 2024-04-14T13:39:27.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 127102 12.80 MB/sec execute 121 sec latency 63.006 ms 2024-04-14T13:39:27.750 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:27 smithi008 ceph-mon[24786]: Standby manager daemon smithi008.yemlhr restarted 2024-04-14T13:39:27.750 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:27 smithi008 ceph-mon[24786]: Standby manager daemon smithi008.yemlhr started 2024-04-14T13:39:27.750 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:27 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.yemlhr/crt"}]: dispatch 2024-04-14T13:39:27.750 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:27 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T13:39:27.751 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:27 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.yemlhr/key"}]: dispatch 2024-04-14T13:39:27.751 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:27 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T13:39:27.751 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:27 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:27.751 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:27 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:27.928 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:27 smithi012 ceph-mon[27236]: Standby manager daemon smithi008.yemlhr restarted 2024-04-14T13:39:27.928 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:27 smithi012 ceph-mon[27236]: Standby manager daemon smithi008.yemlhr started 2024-04-14T13:39:27.928 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:27 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.yemlhr/crt"}]: dispatch 2024-04-14T13:39:27.928 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:27 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T13:39:27.929 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:27 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.yemlhr/key"}]: dispatch 2024-04-14T13:39:27.929 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:27 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T13:39:27.929 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:27 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:27.929 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:27 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:28.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 127400 12.74 MB/sec execute 122 sec latency 43.312 ms 2024-04-14T13:39:28.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:28 smithi008 ceph-mon[24786]: mgrmap e28: smithi012.ksvnxm(active, since 116s), standbys: smithi008.yemlhr 2024-04-14T13:39:28.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:28 smithi008 ceph-mon[24786]: pgmap v61: 97 pgs: 97 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.3 MiB/s wr, 532 op/s 2024-04-14T13:39:29.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:28 smithi012 ceph-mon[27236]: mgrmap e28: smithi012.ksvnxm(active, since 116s), standbys: smithi008.yemlhr 2024-04-14T13:39:29.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:28 smithi012 ceph-mon[27236]: pgmap v61: 97 pgs: 97 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.3 MiB/s wr, 532 op/s 2024-04-14T13:39:29.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 127778 12.79 MB/sec execute 123 sec latency 44.535 ms 2024-04-14T13:39:30.062 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:29 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:30.063 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:29 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:30.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:29 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:30.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:29 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:30.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 128129 12.78 MB/sec execute 124 sec latency 41.674 ms 2024-04-14T13:39:31.079 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:30 smithi008 ceph-mon[24786]: pgmap v62: 97 pgs: 97 active+clean; 332 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 726 op/s 2024-04-14T13:39:31.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:30 smithi012 ceph-mon[27236]: pgmap v62: 97 pgs: 97 active+clean; 332 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 726 op/s 2024-04-14T13:39:31.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 128526 12.73 MB/sec execute 125 sec latency 52.779 ms 2024-04-14T13:39:32.334 INFO:teuthology.orchestra.run.smithi008.stdout: 5 128932 12.68 MB/sec execute 126 sec latency 32.599 ms 2024-04-14T13:39:33.095 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:32 smithi008 ceph-mon[24786]: pgmap v63: 97 pgs: 97 active+clean; 332 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.5 MiB/s wr, 438 op/s 2024-04-14T13:39:33.095 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:32 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:39:33.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:32 smithi012 ceph-mon[27236]: pgmap v63: 97 pgs: 97 active+clean; 332 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.5 MiB/s wr, 438 op/s 2024-04-14T13:39:33.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:32 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:39:33.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 129349 12.68 MB/sec execute 127 sec latency 79.194 ms 2024-04-14T13:39:34.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 129759 12.63 MB/sec execute 128 sec latency 34.582 ms 2024-04-14T13:39:35.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 130058 12.62 MB/sec execute 129 sec latency 57.362 ms 2024-04-14T13:39:35.439 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:35 smithi008 ceph-mon[24786]: pgmap v64: 97 pgs: 97 active+clean; 333 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 MiB/s wr, 675 op/s 2024-04-14T13:39:35.439 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:35 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:35.439 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:35 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:35.439 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:35 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:39:35.439 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:35 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:39:35.439 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:35 smithi008 ceph-mon[24786]: pgmap v65: 97 pgs: 97 active+clean; 333 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.9 MiB/s wr, 486 op/s 2024-04-14T13:39:35.439 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:35 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:35.439 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:35 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:35.439 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:35 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:39:35.439 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:35 smithi008 ceph-mon[24786]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mgr fail", "who": "smithi012.ksvnxm"}]: dispatch 2024-04-14T13:39:35.440 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:35 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mgr fail", "who": "smithi012.ksvnxm"}]: dispatch 2024-04-14T13:39:35.440 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:35 smithi008 ceph-mon[24786]: osdmap e72: 8 total, 8 up, 8 in 2024-04-14T13:39:35.440 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:35 smithi008 ceph-mon[24786]: Standby manager daemon smithi012.ksvnxm started 2024-04-14T13:39:35.783 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:35 smithi012 ceph-mon[27236]: pgmap v64: 97 pgs: 97 active+clean; 333 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 MiB/s wr, 675 op/s 2024-04-14T13:39:35.783 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:35 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:35.783 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:35 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:35.784 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:35 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:39:35.784 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:35 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:39:35.784 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:35 smithi012 ceph-mon[27236]: pgmap v65: 97 pgs: 97 active+clean; 333 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.9 MiB/s wr, 486 op/s 2024-04-14T13:39:35.784 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:35 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:35.784 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:35 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' 2024-04-14T13:39:35.784 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:35 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:39:35.784 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:35 smithi012 ceph-mon[27236]: from='mgr.24471 172.21.15.12:0/3381566663' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mgr fail", "who": "smithi012.ksvnxm"}]: dispatch 2024-04-14T13:39:35.784 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:35 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "mgr fail", "who": "smithi012.ksvnxm"}]: dispatch 2024-04-14T13:39:35.784 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:35 smithi012 ceph-mon[27236]: osdmap e72: 8 total, 8 up, 8 in 2024-04-14T13:39:35.784 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:35 smithi012 ceph-mon[27236]: Standby manager daemon smithi012.ksvnxm started 2024-04-14T13:39:36.333 INFO:teuthology.orchestra.run.smithi008.stdout: 5 130309 12.55 MB/sec execute 130 sec latency 68.905 ms 2024-04-14T13:39:36.465 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T13:39:36.465 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd='[{"prefix": "mgr fail", "who": "smithi012.ksvnxm"}]': finished 2024-04-14T13:39:36.465 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: mgrmap e29: smithi008.yemlhr(active, starting, since 0.863424s), standbys: smithi012.ksvnxm 2024-04-14T13:39:36.465 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:39:36.465 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:39:36.465 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wqbvfi"}]: dispatch 2024-04-14T13:39:36.465 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi012.vjovzp"}]: dispatch 2024-04-14T13:39:36.465 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr metadata", "who": "smithi008.yemlhr", "id": "smithi008.yemlhr"}]: dispatch 2024-04-14T13:39:36.465 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr metadata", "who": "smithi012.ksvnxm", "id": "smithi012.ksvnxm"}]: dispatch 2024-04-14T13:39:36.465 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:39:36.466 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:39:36.466 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:39:36.466 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:39:36.466 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:39:36.466 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:39:36.466 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:39:36.466 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:39:36.466 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T13:39:36.466 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T13:39:36.466 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T13:39:36.466 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: Manager daemon smithi008.yemlhr is now available 2024-04-14T13:39:36.466 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:39:36.466 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:36.466 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:39:36.466 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:39:36.467 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:36 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:39:36.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T13:39:36.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.24471 ' entity='mgr.smithi012.ksvnxm' cmd='[{"prefix": "mgr fail", "who": "smithi012.ksvnxm"}]': finished 2024-04-14T13:39:36.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: mgrmap e29: smithi008.yemlhr(active, starting, since 0.863424s), standbys: smithi012.ksvnxm 2024-04-14T13:39:36.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:39:36.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:39:36.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wqbvfi"}]: dispatch 2024-04-14T13:39:36.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi012.vjovzp"}]: dispatch 2024-04-14T13:39:36.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr metadata", "who": "smithi008.yemlhr", "id": "smithi008.yemlhr"}]: dispatch 2024-04-14T13:39:36.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr metadata", "who": "smithi012.ksvnxm", "id": "smithi012.ksvnxm"}]: dispatch 2024-04-14T13:39:36.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:39:36.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:39:36.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:39:36.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:39:36.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:39:36.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:39:36.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:39:36.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:39:36.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T13:39:36.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T13:39:36.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T13:39:36.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: Manager daemon smithi008.yemlhr is now available 2024-04-14T13:39:36.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:39:36.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:36.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:39:36.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:39:36.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:36 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:39:37.334 INFO:teuthology.orchestra.run.smithi008.stdout: 5 130598 12.53 MB/sec execute 131 sec latency 80.001 ms 2024-04-14T13:39:37.506 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:37 smithi008 ceph-mon[24786]: [14/Apr/2024:13:39:36] ENGINE Bus STARTING 2024-04-14T13:39:37.506 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:37 smithi008 ceph-mon[24786]: [14/Apr/2024:13:39:36] ENGINE Serving on https://172.21.15.8:7150 2024-04-14T13:39:37.506 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:37 smithi008 ceph-mon[24786]: [14/Apr/2024:13:39:36] ENGINE Serving on http://172.21.15.8:8765 2024-04-14T13:39:37.506 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:37 smithi008 ceph-mon[24786]: [14/Apr/2024:13:39:36] ENGINE Bus STARTED 2024-04-14T13:39:37.506 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:37 smithi008 ceph-mon[24786]: mgrmap e30: smithi008.yemlhr(active, since 1.8675s), standbys: smithi012.ksvnxm 2024-04-14T13:39:37.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:37 smithi008 ceph-mon[24786]: pgmap v3: 97 pgs: 97 active+clean; 327 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-14T13:39:37.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:37 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:37.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:37 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:37.705 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:37 smithi012 ceph-mon[27236]: [14/Apr/2024:13:39:36] ENGINE Bus STARTING 2024-04-14T13:39:37.705 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:37 smithi012 ceph-mon[27236]: [14/Apr/2024:13:39:36] ENGINE Serving on https://172.21.15.8:7150 2024-04-14T13:39:37.705 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:37 smithi012 ceph-mon[27236]: [14/Apr/2024:13:39:36] ENGINE Serving on http://172.21.15.8:8765 2024-04-14T13:39:37.705 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:37 smithi012 ceph-mon[27236]: [14/Apr/2024:13:39:36] ENGINE Bus STARTED 2024-04-14T13:39:37.705 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:37 smithi012 ceph-mon[27236]: mgrmap e30: smithi008.yemlhr(active, since 1.8675s), standbys: smithi012.ksvnxm 2024-04-14T13:39:37.705 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:37 smithi012 ceph-mon[27236]: pgmap v3: 97 pgs: 97 active+clean; 327 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-14T13:39:37.705 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:37 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:37.705 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:37 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:38.334 INFO:teuthology.orchestra.run.smithi008.stdout: 5 130944 12.50 MB/sec execute 132 sec latency 55.900 ms 2024-04-14T13:39:38.527 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:38 smithi008 ceph-mon[24786]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T13:39:38.527 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:38 smithi008 ceph-mon[24786]: Cluster is now healthy 2024-04-14T13:39:38.527 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:38 smithi008 ceph-mon[24786]: pgmap v4: 97 pgs: 97 active+clean; 327 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-14T13:39:38.527 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:38 smithi008 ceph-mon[24786]: mgrmap e31: smithi008.yemlhr(active, since 3s), standbys: smithi012.ksvnxm 2024-04-14T13:39:38.609 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:38 smithi012 ceph-mon[27236]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T13:39:38.609 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:38 smithi012 ceph-mon[27236]: Cluster is now healthy 2024-04-14T13:39:38.609 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:38 smithi012 ceph-mon[27236]: pgmap v4: 97 pgs: 97 active+clean; 327 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-14T13:39:38.609 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:38 smithi012 ceph-mon[27236]: mgrmap e31: smithi008.yemlhr(active, since 3s), standbys: smithi012.ksvnxm 2024-04-14T13:39:39.334 INFO:teuthology.orchestra.run.smithi008.stdout: 5 131325 12.55 MB/sec execute 133 sec latency 38.864 ms 2024-04-14T13:39:40.334 INFO:teuthology.orchestra.run.smithi008.stdout: 5 131648 12.52 MB/sec execute 134 sec latency 39.135 ms 2024-04-14T13:39:40.550 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:40 smithi008 ceph-mon[24786]: pgmap v5: 97 pgs: 97 active+clean; 327 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-14T13:39:40.550 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:40 smithi008 ceph-mon[24786]: mgrmap e32: smithi008.yemlhr(active, since 4s), standbys: smithi012.ksvnxm 2024-04-14T13:39:40.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:40 smithi012 ceph-mon[27236]: pgmap v5: 97 pgs: 97 active+clean; 327 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-14T13:39:40.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:40 smithi012 ceph-mon[27236]: mgrmap e32: smithi008.yemlhr(active, since 4s), standbys: smithi012.ksvnxm 2024-04-14T13:39:41.334 INFO:teuthology.orchestra.run.smithi008.stdout: 5 132083 12.50 MB/sec execute 135 sec latency 51.672 ms 2024-04-14T13:39:42.070 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:41 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:42.070 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:41 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:42.070 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:41 smithi008 ceph-mon[24786]: pgmap v6: 97 pgs: 97 active+clean; 328 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 2.0 MiB/s wr, 71 op/s 2024-04-14T13:39:42.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:41 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:42.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:41 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:42.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:41 smithi012 ceph-mon[27236]: pgmap v6: 97 pgs: 97 active+clean; 328 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 2.0 MiB/s wr, 71 op/s 2024-04-14T13:39:42.334 INFO:teuthology.orchestra.run.smithi008.stdout: 5 132402 12.44 MB/sec execute 136 sec latency 37.332 ms 2024-04-14T13:39:43.334 INFO:teuthology.orchestra.run.smithi008.stdout: 5 132739 12.44 MB/sec execute 137 sec latency 52.674 ms 2024-04-14T13:39:43.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:43 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:43.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:43 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:43.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:43 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:43.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:43 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:44.334 INFO:teuthology.orchestra.run.smithi008.stdout: 5 132935 12.37 MB/sec execute 138 sec latency 82.108 ms 2024-04-14T13:39:44.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:44 smithi008 ceph-mon[24786]: pgmap v7: 97 pgs: 97 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 34 KiB/s rd, 4.2 MiB/s wr, 381 op/s 2024-04-14T13:39:44.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:44 smithi012 ceph-mon[27236]: pgmap v7: 97 pgs: 97 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 34 KiB/s rd, 4.2 MiB/s wr, 381 op/s 2024-04-14T13:39:45.334 INFO:teuthology.orchestra.run.smithi008.stdout: 5 133262 12.31 MB/sec execute 139 sec latency 61.515 ms 2024-04-14T13:39:45.686 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:45 smithi008 ceph-mon[24786]: Standby manager daemon smithi012.ksvnxm restarted 2024-04-14T13:39:45.686 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:45 smithi008 ceph-mon[24786]: Standby manager daemon smithi012.ksvnxm started 2024-04-14T13:39:45.686 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:45 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.12:0/21935758' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi012.ksvnxm/crt"}]: dispatch 2024-04-14T13:39:45.686 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:45 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.12:0/21935758' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T13:39:45.687 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:45 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.12:0/21935758' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi012.ksvnxm/key"}]: dispatch 2024-04-14T13:39:45.687 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:45 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.12:0/21935758' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T13:39:45.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:45 smithi012 ceph-mon[27236]: Standby manager daemon smithi012.ksvnxm restarted 2024-04-14T13:39:45.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:45 smithi012 ceph-mon[27236]: Standby manager daemon smithi012.ksvnxm started 2024-04-14T13:39:45.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:45 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.12:0/21935758' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi012.ksvnxm/crt"}]: dispatch 2024-04-14T13:39:45.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:45 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.12:0/21935758' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T13:39:45.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:45 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.12:0/21935758' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi012.ksvnxm/key"}]: dispatch 2024-04-14T13:39:45.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:45 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.12:0/21935758' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T13:39:46.334 INFO:teuthology.orchestra.run.smithi008.stdout: 5 133566 12.31 MB/sec execute 140 sec latency 80.234 ms 2024-04-14T13:39:46.526 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:46 smithi008 ceph-mon[24786]: pgmap v8: 97 pgs: 97 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 3.3 MiB/s wr, 296 op/s 2024-04-14T13:39:46.526 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:46 smithi008 ceph-mon[24786]: mgrmap e33: smithi008.yemlhr(active, since 10s), standbys: smithi012.ksvnxm 2024-04-14T13:39:46.526 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:46 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:46.526 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:46 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:46.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:46 smithi012 ceph-mon[27236]: pgmap v8: 97 pgs: 97 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 3.3 MiB/s wr, 296 op/s 2024-04-14T13:39:46.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:46 smithi012 ceph-mon[27236]: mgrmap e33: smithi008.yemlhr(active, since 10s), standbys: smithi012.ksvnxm 2024-04-14T13:39:46.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:46 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:46.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:46 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:47.334 INFO:teuthology.orchestra.run.smithi008.stdout: 5 133747 12.25 MB/sec execute 141 sec latency 85.067 ms 2024-04-14T13:39:48.334 INFO:teuthology.orchestra.run.smithi008.stdout: 5 134217 12.25 MB/sec execute 142 sec latency 48.496 ms 2024-04-14T13:39:48.805 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:48 smithi008 ceph-mon[24786]: pgmap v9: 97 pgs: 97 active+clean; 326 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.4 MiB/s wr, 445 op/s 2024-04-14T13:39:48.806 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:48 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:48.806 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:48 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:49.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:48 smithi012 ceph-mon[27236]: pgmap v9: 97 pgs: 97 active+clean; 326 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.4 MiB/s wr, 445 op/s 2024-04-14T13:39:49.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:48 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:49.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:48 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:49.323 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:39:49.334 INFO:teuthology.orchestra.run.smithi008.stdout: 5 134760 12.31 MB/sec execute 143 sec latency 36.211 ms 2024-04-14T13:39:50.334 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:39:50.335 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (102s) 3s ago 10m 17.1M - 0.23.0 ba2b418f427c cfddb9a7f858 2024-04-14T13:39:50.335 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (10m) 3s ago 10m 6405k - 17.2.0 e1d6a67b021e 68b810a9ff93 2024-04-14T13:39:50.335 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (10m) 9s ago 10m 6400k - 17.2.0 e1d6a67b021e 6bc31c51b938 2024-04-14T13:39:50.335 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 unknown 3s ago 10m - - 2024-04-14T13:39:50.335 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (4m) 3s ago 4m 37.4M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:39:50.335 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (4m) 9s ago 4m 13.4M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:39:50.335 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (34s) 3s ago 12m 485M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:39:50.335 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283 running (2m) 9s ago 10m 99.5M - 19.0.0-2778-ga971fd5f b96c509eef84 089c1c1b8529 2024-04-14T13:39:50.335 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (12m) 3s ago 12m 60.9M 2048M 17.2.0 e1d6a67b021e 7786c27a60d0 2024-04-14T13:39:50.335 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (9m) 9s ago 9m 42.7M 2048M 17.2.0 e1d6a67b021e f6cd483dfa31 2024-04-14T13:39:50.335 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (4m) 3s ago 4m 266M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:39:50.335 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (4m) 9s ago 4m 84.2M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:39:50.335 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (75s) 3s ago 10m 8963k - 1.3.1 1dbe0e931976 646dfca70be4 2024-04-14T13:39:50.335 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (46s) 9s ago 9m 11.0M - 1.3.1 1dbe0e931976 0a60dcd4044c 2024-04-14T13:39:50.335 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (9m) 3s ago 9m 475M 1937M 17.2.0 e1d6a67b021e 3b8cdea7890c 2024-04-14T13:39:50.336 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (8m) 3s ago 8m 406M 1937M 17.2.0 e1d6a67b021e be86642c7790 2024-04-14T13:39:50.336 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (8m) 3s ago 8m 274M 1937M 17.2.0 e1d6a67b021e b1017f00578f 2024-04-14T13:39:50.336 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (7m) 3s ago 7m 434M 1937M 17.2.0 e1d6a67b021e 66fca4b191be 2024-04-14T13:39:50.336 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (7m) 9s ago 7m 486M 2705M 17.2.0 e1d6a67b021e 43b02cb64951 2024-04-14T13:39:50.336 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (7m) 9s ago 7m 494M 2705M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:39:50.336 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (6m) 9s ago 6m 415M 2705M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:39:50.336 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (6m) 9s ago 6m 583M 2705M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:39:50.336 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (49s) 3s ago 9m 53.3M - 2.33.4 514e6a882f6e ce1a0218005a 2024-04-14T13:39:50.336 INFO:teuthology.orchestra.run.smithi008.stdout: 5 135128 12.31 MB/sec execute 144 sec latency 33.631 ms 2024-04-14T13:39:50.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:50 smithi012 ceph-mon[27236]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:39:50.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:50 smithi012 ceph-mon[27236]: pgmap v10: 97 pgs: 97 active+clean; 326 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.1 MiB/s wr, 407 op/s 2024-04-14T13:39:50.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:50 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:50.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:50 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:39:50.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:50 smithi008 ceph-mon[24786]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:39:50.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:50 smithi008 ceph-mon[24786]: pgmap v10: 97 pgs: 97 active+clean; 326 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.1 MiB/s wr, 407 op/s 2024-04-14T13:39:50.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:50 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:50.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:50 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:39:50.961 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:39:50.961 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:39:50.962 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:39:50.962 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:39:50.962 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:39:50.962 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:39:50.962 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:39:50.962 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:39:50.962 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T13:39:50.962 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:39:50.962 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:39:50.962 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:39:50.962 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:39:50.962 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:39:50.962 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-14T13:39:50.962 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:39:50.962 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:39:50.962 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:39:51.334 INFO:teuthology.orchestra.run.smithi008.stdout: 5 135465 12.28 MB/sec execute 145 sec latency 56.521 ms 2024-04-14T13:39:51.479 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:39:51.480 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:39:51.480 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:39:51.480 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:39:51.480 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-14T13:39:51.480 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-14T13:39:51.480 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:39:51.480 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "2/23 daemons upgraded", 2024-04-14T13:39:51.480 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "", 2024-04-14T13:39:51.480 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:39:51.480 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:39:51.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:51 smithi008 ceph-mon[24786]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:39:51.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:51 smithi008 ceph-mon[24786]: from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:39:51.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:51 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/1112991463' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:39:52.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:51 smithi012 ceph-mon[27236]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:39:52.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:51 smithi012 ceph-mon[27236]: from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:39:52.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:51 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/1112991463' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:39:52.100 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-04-14T13:39:52.335 INFO:teuthology.orchestra.run.smithi008.stdout: 5 135807 12.23 MB/sec execute 146 sec latency 43.964 ms 2024-04-14T13:39:52.889 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:52 smithi008 ceph-mon[24786]: from='client.14854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:39:52.889 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:52 smithi008 ceph-mon[24786]: pgmap v11: 97 pgs: 97 active+clean; 330 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.9 MiB/s wr, 440 op/s 2024-04-14T13:39:52.889 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:52 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/1350880256' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:39:53.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:52 smithi012 ceph-mon[27236]: from='client.14854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:39:53.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:52 smithi012 ceph-mon[27236]: pgmap v11: 97 pgs: 97 active+clean; 330 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.9 MiB/s wr, 440 op/s 2024-04-14T13:39:53.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:52 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/1350880256' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:39:53.335 INFO:teuthology.orchestra.run.smithi008.stdout: 5 136121 12.18 MB/sec execute 147 sec latency 35.729 ms 2024-04-14T13:39:54.335 INFO:teuthology.orchestra.run.smithi008.stdout: 5 136482 12.18 MB/sec execute 148 sec latency 54.125 ms 2024-04-14T13:39:54.649 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:54 smithi008 ceph-mon[24786]: pgmap v12: 97 pgs: 97 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 5.8 MiB/s wr, 605 op/s 2024-04-14T13:39:55.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:54 smithi012 ceph-mon[27236]: pgmap v12: 97 pgs: 97 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 5.8 MiB/s wr, 605 op/s 2024-04-14T13:39:55.335 INFO:teuthology.orchestra.run.smithi008.stdout: 5 136915 12.15 MB/sec execute 149 sec latency 54.934 ms 2024-04-14T13:39:55.922 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:55 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:55.922 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:55 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:55.923 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:55 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:39:55.923 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:55 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:39:55.923 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:55 smithi008 ceph-mon[24786]: Updating smithi008:/etc/ceph/ceph.conf 2024-04-14T13:39:55.923 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:55 smithi008 ceph-mon[24786]: Updating smithi012:/etc/ceph/ceph.conf 2024-04-14T13:39:56.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:55 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:56.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:55 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:56.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:55 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:39:56.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:55 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:39:56.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:55 smithi012 ceph-mon[27236]: Updating smithi008:/etc/ceph/ceph.conf 2024-04-14T13:39:56.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:55 smithi012 ceph-mon[27236]: Updating smithi012:/etc/ceph/ceph.conf 2024-04-14T13:39:56.335 INFO:teuthology.orchestra.run.smithi008.stdout: 5 137218 12.14 MB/sec execute 150 sec latency 60.066 ms 2024-04-14T13:39:56.689 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:56 smithi008 ceph-mon[24786]: Updating smithi012:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T13:39:56.689 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:56 smithi008 ceph-mon[24786]: Updating smithi008:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T13:39:56.689 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:56 smithi008 ceph-mon[24786]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:39:56.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:56 smithi008 ceph-mon[24786]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:39:56.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:56 smithi008 ceph-mon[24786]: pgmap v13: 97 pgs: 97 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.2 MiB/s wr, 413 op/s 2024-04-14T13:39:56.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:56 smithi008 ceph-mon[24786]: Updating smithi012:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T13:39:56.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:56 smithi008 ceph-mon[24786]: Updating smithi008:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T13:39:56.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:56 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:56.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:56 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:56.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:56 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:56.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:56 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:56.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:56 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:56.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:56 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:57.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:56 smithi012 ceph-mon[27236]: Updating smithi012:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T13:39:57.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:56 smithi012 ceph-mon[27236]: Updating smithi008:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T13:39:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:56 smithi012 ceph-mon[27236]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:39:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:56 smithi012 ceph-mon[27236]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:39:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:56 smithi012 ceph-mon[27236]: pgmap v13: 97 pgs: 97 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.2 MiB/s wr, 413 op/s 2024-04-14T13:39:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:56 smithi012 ceph-mon[27236]: Updating smithi012:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T13:39:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:56 smithi012 ceph-mon[27236]: Updating smithi008:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T13:39:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:56 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:56 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:56 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:56 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:56 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:56 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:39:57.335 INFO:teuthology.orchestra.run.smithi008.stdout: 5 137840 12.16 MB/sec execute 151 sec latency 45.295 ms 2024-04-14T13:39:57.696 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:57 smithi008 ceph-mon[24786]: pgmap v14: 97 pgs: 97 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.8 MiB/s wr, 471 op/s 2024-04-14T13:39:57.696 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:57 smithi008 ceph-mon[24786]: Reconfiguring prometheus.smithi008 (dependencies changed)... 2024-04-14T13:39:57.696 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:57 smithi008 ceph-mon[24786]: Reconfiguring daemon prometheus.smithi008 on smithi008 2024-04-14T13:39:57.696 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:57 smithi008 ceph-mon[24786]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T13:39:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:57 smithi012 ceph-mon[27236]: pgmap v14: 97 pgs: 97 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.8 MiB/s wr, 471 op/s 2024-04-14T13:39:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:57 smithi012 ceph-mon[27236]: Reconfiguring prometheus.smithi008 (dependencies changed)... 2024-04-14T13:39:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:57 smithi012 ceph-mon[27236]: Reconfiguring daemon prometheus.smithi008 on smithi008 2024-04-14T13:39:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:57 smithi012 ceph-mon[27236]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T13:39:58.335 INFO:teuthology.orchestra.run.smithi008.stdout: 5 138470 12.25 MB/sec execute 152 sec latency 28.465 ms 2024-04-14T13:39:59.335 INFO:teuthology.orchestra.run.smithi008.stdout: 5 139254 12.28 MB/sec execute 153 sec latency 17.418 ms 2024-04-14T13:39:59.707 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:39:59 smithi008 ceph-mon[24786]: pgmap v15: 97 pgs: 97 active+clean; 328 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.2 MiB/s wr, 515 op/s 2024-04-14T13:40:00.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:39:59 smithi012 ceph-mon[27236]: pgmap v15: 97 pgs: 97 active+clean; 328 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.2 MiB/s wr, 515 op/s 2024-04-14T13:40:00.336 INFO:teuthology.orchestra.run.smithi008.stdout: 5 139940 12.31 MB/sec execute 154 sec latency 28.277 ms 2024-04-14T13:40:00.730 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:00 smithi008 ceph-mon[24786]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T13:40:00.730 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:00 smithi008 ceph-mon[24786]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:40:00.730 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:00 smithi008 ceph-mon[24786]: daemon grafana.smithi008 on smithi008 is in unknown state 2024-04-14T13:40:01.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:00 smithi012 ceph-mon[27236]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T13:40:01.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:00 smithi012 ceph-mon[27236]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:40:01.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:00 smithi012 ceph-mon[27236]: daemon grafana.smithi008 on smithi008 is in unknown state 2024-04-14T13:40:01.336 INFO:teuthology.orchestra.run.smithi008.stdout: 5 140644 12.35 MB/sec execute 155 sec latency 35.071 ms 2024-04-14T13:40:01.746 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:01 smithi008 ceph-mon[24786]: pgmap v16: 97 pgs: 97 active+clean; 328 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.2 MiB/s wr, 515 op/s 2024-04-14T13:40:02.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:01 smithi012 ceph-mon[27236]: pgmap v16: 97 pgs: 97 active+clean; 328 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.2 MiB/s wr, 515 op/s 2024-04-14T13:40:02.336 INFO:teuthology.orchestra.run.smithi008.stdout: 5 141262 12.37 MB/sec execute 156 sec latency 25.130 ms 2024-04-14T13:40:03.336 INFO:teuthology.orchestra.run.smithi008.stdout: 5 141874 12.43 MB/sec execute 157 sec latency 38.838 ms 2024-04-14T13:40:03.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:03 smithi008 ceph-mon[24786]: pgmap v17: 97 pgs: 97 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 8.9 MiB/s wr, 928 op/s 2024-04-14T13:40:04.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:03 smithi012 ceph-mon[27236]: pgmap v17: 97 pgs: 97 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 8.9 MiB/s wr, 928 op/s 2024-04-14T13:40:04.336 INFO:teuthology.orchestra.run.smithi008.stdout: 5 142396 12.45 MB/sec execute 158 sec latency 28.200 ms 2024-04-14T13:40:05.336 INFO:teuthology.orchestra.run.smithi008.stdout: 5 142863 12.43 MB/sec execute 159 sec latency 34.040 ms 2024-04-14T13:40:05.737 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:05 smithi008 ceph-mon[24786]: pgmap v18: 97 pgs: 97 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 7.0 MiB/s wr, 706 op/s 2024-04-14T13:40:05.737 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:05 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:05.737 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:05 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:40:06.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:05 smithi012 ceph-mon[27236]: pgmap v18: 97 pgs: 97 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 7.0 MiB/s wr, 706 op/s 2024-04-14T13:40:06.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:05 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:06.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:05 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:40:06.336 INFO:teuthology.orchestra.run.smithi008.stdout: 5 143305 12.42 MB/sec execute 160 sec latency 100.685 ms 2024-04-14T13:40:07.336 INFO:teuthology.orchestra.run.smithi008.stdout: 5 143820 12.40 MB/sec execute 161 sec latency 57.888 ms 2024-04-14T13:40:07.844 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:07 smithi008 ceph-mon[24786]: pgmap v19: 97 pgs: 97 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 7.0 MiB/s wr, 706 op/s 2024-04-14T13:40:08.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:07 smithi012 ceph-mon[27236]: pgmap v19: 97 pgs: 97 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 7.0 MiB/s wr, 706 op/s 2024-04-14T13:40:08.336 INFO:teuthology.orchestra.run.smithi008.stdout: 5 144309 12.42 MB/sec execute 162 sec latency 70.058 ms 2024-04-14T13:40:09.336 INFO:teuthology.orchestra.run.smithi008.stdout: 5 144989 12.45 MB/sec execute 163 sec latency 21.981 ms 2024-04-14T13:40:09.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:09 smithi008 ceph-mon[24786]: pgmap v20: 97 pgs: 97 active+clean; 343 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 932 op/s 2024-04-14T13:40:09.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:09 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:09.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:09 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:09.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:09 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:40:09.863 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:09 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:40:10.036 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:09 smithi012 ceph-mon[27236]: pgmap v20: 97 pgs: 97 active+clean; 343 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 932 op/s 2024-04-14T13:40:10.037 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:09 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:10.037 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:09 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:10.037 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:09 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:40:10.037 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:09 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:40:10.336 INFO:teuthology.orchestra.run.smithi008.stdout: 5 145561 12.51 MB/sec execute 164 sec latency 38.054 ms 2024-04-14T13:40:10.955 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:10 smithi008 ceph-mon[24786]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:40:11.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:10 smithi012 ceph-mon[27236]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:40:11.336 INFO:teuthology.orchestra.run.smithi008.stdout: 5 145896 12.50 MB/sec execute 165 sec latency 57.697 ms 2024-04-14T13:40:11.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:11 smithi008 ceph-mon[24786]: pgmap v21: 97 pgs: 97 active+clean; 343 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 7.5 MiB/s wr, 708 op/s 2024-04-14T13:40:11.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:11 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi012.ksvnxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:40:11.726 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:11 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:40:11.726 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:11 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:40:11.985 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:11 smithi012 ceph-mon[27236]: pgmap v21: 97 pgs: 97 active+clean; 343 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 7.5 MiB/s wr, 708 op/s 2024-04-14T13:40:11.985 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:11 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi012.ksvnxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:40:11.985 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:11 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:40:11.986 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:11 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:40:12.336 INFO:teuthology.orchestra.run.smithi008.stdout: 5 146252 12.46 MB/sec execute 166 sec latency 58.022 ms 2024-04-14T13:40:12.742 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:12 smithi008 ceph-mon[24786]: Upgrade: Updating mgr.smithi012.ksvnxm 2024-04-14T13:40:12.742 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:12 smithi008 ceph-mon[24786]: Deploying daemon mgr.smithi012.ksvnxm on smithi012 2024-04-14T13:40:13.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:12 smithi012 ceph-mon[27236]: Upgrade: Updating mgr.smithi012.ksvnxm 2024-04-14T13:40:13.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:12 smithi012 ceph-mon[27236]: Deploying daemon mgr.smithi012.ksvnxm on smithi012 2024-04-14T13:40:13.336 INFO:teuthology.orchestra.run.smithi008.stdout: 5 146541 12.42 MB/sec execute 167 sec latency 103.684 ms 2024-04-14T13:40:13.751 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:13 smithi008 ceph-mon[24786]: pgmap v22: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-04-14T13:40:13.983 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:13 smithi012 ceph-mon[27236]: pgmap v22: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-04-14T13:40:14.336 INFO:teuthology.orchestra.run.smithi008.stdout: 5 147136 12.44 MB/sec execute 168 sec latency 34.091 ms 2024-04-14T13:40:15.336 INFO:teuthology.orchestra.run.smithi008.stdout: 5 147741 12.45 MB/sec execute 169 sec latency 21.160 ms 2024-04-14T13:40:15.767 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:15 smithi008 ceph-mon[24786]: pgmap v23: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.6 MiB/s wr, 615 op/s 2024-04-14T13:40:15.778 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:15 smithi012 ceph-mon[27236]: pgmap v23: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.6 MiB/s wr, 615 op/s 2024-04-14T13:40:16.336 INFO:teuthology.orchestra.run.smithi008.stdout: 5 148280 12.47 MB/sec execute 170 sec latency 83.418 ms 2024-04-14T13:40:17.281 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:17 smithi008 ceph-mon[24786]: pgmap v24: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.6 MiB/s wr, 615 op/s 2024-04-14T13:40:17.281 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:17 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:17.281 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:17 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:17.281 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:17 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:40:17.336 INFO:teuthology.orchestra.run.smithi008.stdout: 5 148708 12.46 MB/sec execute 171 sec latency 30.658 ms 2024-04-14T13:40:17.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:17 smithi012 ceph-mon[27236]: pgmap v24: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.6 MiB/s wr, 615 op/s 2024-04-14T13:40:17.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:17 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:17.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:17 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:17.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:17 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:40:18.337 INFO:teuthology.orchestra.run.smithi008.stdout: 5 149116 12.48 MB/sec execute 172 sec latency 69.898 ms 2024-04-14T13:40:19.319 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:19 smithi012 ceph-mon[27236]: pgmap v25: 97 pgs: 97 active+clean; 333 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 908 op/s 2024-04-14T13:40:19.328 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:19 smithi008 ceph-mon[24786]: pgmap v25: 97 pgs: 97 active+clean; 333 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 908 op/s 2024-04-14T13:40:19.338 INFO:teuthology.orchestra.run.smithi008.stdout: 5 149538 12.48 MB/sec execute 173 sec latency 48.227 ms 2024-04-14T13:40:20.337 INFO:teuthology.orchestra.run.smithi008.stdout: 5 149933 12.46 MB/sec execute 174 sec latency 35.992 ms 2024-04-14T13:40:21.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:21 smithi012 ceph-mon[27236]: pgmap v26: 97 pgs: 97 active+clean; 333 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.9 MiB/s wr, 595 op/s 2024-04-14T13:40:21.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:21 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:40:21.337 INFO:teuthology.orchestra.run.smithi008.stdout: 5 150150 12.40 MB/sec execute 175 sec latency 59.151 ms 2024-04-14T13:40:21.368 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:21 smithi008 ceph-mon[24786]: pgmap v26: 97 pgs: 97 active+clean; 333 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.9 MiB/s wr, 595 op/s 2024-04-14T13:40:21.368 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:21 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:40:22.337 INFO:teuthology.orchestra.run.smithi008.stdout: 5 150438 12.39 MB/sec execute 176 sec latency 70.011 ms 2024-04-14T13:40:22.574 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:40:22.905 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:22 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:22.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:22 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:22.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:22 smithi008 ceph-mon[24786]: pgmap v27: 97 pgs: 97 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 829 op/s 2024-04-14T13:40:22.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:22 smithi008 ceph-mon[24786]: from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:40:23.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:22 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:23.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:22 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:23.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:22 smithi012 ceph-mon[27236]: pgmap v27: 97 pgs: 97 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 829 op/s 2024-04-14T13:40:23.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:22 smithi012 ceph-mon[27236]: from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:40:23.337 INFO:teuthology.orchestra.run.smithi008.stdout: 5 150723 12.35 MB/sec execute 177 sec latency 72.370 ms 2024-04-14T13:40:23.573 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:40:23.573 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (2m) 37s ago 11m 17.1M - 0.23.0 ba2b418f427c cfddb9a7f858 2024-04-14T13:40:23.573 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (11m) 37s ago 11m 6405k - 17.2.0 e1d6a67b021e 68b810a9ff93 2024-04-14T13:40:23.573 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (10m) 1s ago 10m 6400k - 17.2.0 e1d6a67b021e 6bc31c51b938 2024-04-14T13:40:23.573 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 unknown 37s ago 11m - - 2024-04-14T13:40:23.573 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (5m) 37s ago 5m 37.4M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:40:23.573 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (5m) 1s ago 5m 13.6M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:40:23.573 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (67s) 37s ago 12m 485M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:40:23.573 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (7s) 1s ago 10m 65.9M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:40:23.573 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (12m) 37s ago 12m 60.9M 2048M 17.2.0 e1d6a67b021e 7786c27a60d0 2024-04-14T13:40:23.573 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (10m) 1s ago 10m 43.7M 2048M 17.2.0 e1d6a67b021e f6cd483dfa31 2024-04-14T13:40:23.574 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (5m) 37s ago 5m 266M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:40:23.574 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (5m) 1s ago 5m 87.8M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:40:23.574 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (108s) 37s ago 10m 8963k - 1.3.1 1dbe0e931976 646dfca70be4 2024-04-14T13:40:23.574 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (79s) 1s ago 10m 17.9M - 1.3.1 1dbe0e931976 0a60dcd4044c 2024-04-14T13:40:23.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (9m) 37s ago 9m 475M 1937M 17.2.0 e1d6a67b021e 3b8cdea7890c 2024-04-14T13:40:23.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (9m) 37s ago 9m 406M 1937M 17.2.0 e1d6a67b021e be86642c7790 2024-04-14T13:40:23.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (8m) 37s ago 8m 274M 1937M 17.2.0 e1d6a67b021e b1017f00578f 2024-04-14T13:40:23.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (8m) 37s ago 8m 434M 1937M 17.2.0 e1d6a67b021e 66fca4b191be 2024-04-14T13:40:23.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (8m) 1s ago 8m 518M 2705M 17.2.0 e1d6a67b021e 43b02cb64951 2024-04-14T13:40:23.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (7m) 1s ago 7m 558M 2705M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:40:23.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (7m) 1s ago 7m 486M 2705M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:40:23.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (6m) 1s ago 6m 377M 2705M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:40:23.574 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 starting - - - - 2024-04-14T13:40:24.194 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:40:24.194 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:40:24.194 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:40:24.194 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:40:24.194 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:40:24.194 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:40:24.195 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:40:24.195 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:40:24.195 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T13:40:24.195 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:40:24.195 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:40:24.195 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:40:24.195 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:40:24.195 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:40:24.195 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-14T13:40:24.195 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:40:24.195 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:40:24.195 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:40:24.337 INFO:teuthology.orchestra.run.smithi008.stdout: 5 151221 12.35 MB/sec execute 178 sec latency 33.201 ms 2024-04-14T13:40:24.444 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:24 smithi008 ceph-mon[24786]: from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:40:24.444 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:24 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:24.445 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:24 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:24.445 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:24 smithi008 ceph-mon[24786]: from='client.24625 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:40:24.445 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:24 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/2441592819' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:40:24.457 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:24 smithi012 ceph-mon[27236]: from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:40:24.457 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:24 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:24.457 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:24 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:24.457 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:24 smithi012 ceph-mon[27236]: from='client.24625 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:40:24.457 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:24 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/2441592819' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:40:24.779 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:40:24.779 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:40:24.779 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:40:24.779 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:40:24.779 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-14T13:40:24.779 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-14T13:40:24.779 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:40:24.779 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "2/23 daemons upgraded", 2024-04-14T13:40:24.780 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading mgr daemons", 2024-04-14T13:40:24.780 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:40:24.780 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:40:25.338 INFO:teuthology.orchestra.run.smithi008.stdout: 5 151911 12.39 MB/sec execute 179 sec latency 31.004 ms 2024-04-14T13:40:25.451 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:25 smithi008 ceph-mon[24786]: pgmap v28: 97 pgs: 97 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.3 MiB/s wr, 526 op/s 2024-04-14T13:40:25.451 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:25 smithi008 ceph-mon[24786]: Standby manager daemon smithi012.ksvnxm restarted 2024-04-14T13:40:25.451 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:25 smithi008 ceph-mon[24786]: Standby manager daemon smithi012.ksvnxm started 2024-04-14T13:40:25.451 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:25 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.12:0/833725931' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi012.ksvnxm/crt"}]: dispatch 2024-04-14T13:40:25.451 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:25 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.12:0/833725931' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T13:40:25.452 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:25 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.12:0/833725931' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi012.ksvnxm/key"}]: dispatch 2024-04-14T13:40:25.452 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:25 smithi008 ceph-mon[24786]: from='mgr.? 172.21.15.12:0/833725931' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T13:40:25.486 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T13:40:25.486 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:40:25.486 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in unknown state 2024-04-14T13:40:25.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:25 smithi012 ceph-mon[27236]: pgmap v28: 97 pgs: 97 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.3 MiB/s wr, 526 op/s 2024-04-14T13:40:25.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:25 smithi012 ceph-mon[27236]: Standby manager daemon smithi012.ksvnxm restarted 2024-04-14T13:40:25.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:25 smithi012 ceph-mon[27236]: Standby manager daemon smithi012.ksvnxm started 2024-04-14T13:40:25.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:25 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.12:0/833725931' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi012.ksvnxm/crt"}]: dispatch 2024-04-14T13:40:25.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:25 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.12:0/833725931' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T13:40:25.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:25 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.12:0/833725931' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi012.ksvnxm/key"}]: dispatch 2024-04-14T13:40:25.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:25 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.12:0/833725931' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T13:40:26.337 INFO:teuthology.orchestra.run.smithi008.stdout: 5 152466 12.42 MB/sec execute 180 sec latency 49.449 ms 2024-04-14T13:40:26.469 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:26 smithi008 ceph-mon[24786]: from='client.14882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:40:26.469 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:26 smithi008 ceph-mon[24786]: mgrmap e34: smithi008.yemlhr(active, since 50s), standbys: smithi012.ksvnxm 2024-04-14T13:40:26.469 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:26 smithi008 ceph-mon[24786]: from='client.? 172.21.15.8:0/3988948167' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:40:26.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:26 smithi012 ceph-mon[27236]: from='client.14882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:40:26.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:26 smithi012 ceph-mon[27236]: mgrmap e34: smithi008.yemlhr(active, since 50s), standbys: smithi012.ksvnxm 2024-04-14T13:40:26.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:26 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/3988948167' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:40:27.337 INFO:teuthology.orchestra.run.smithi008.stdout: 5 152977 12.45 MB/sec execute 181 sec latency 30.213 ms 2024-04-14T13:40:27.665 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:27 smithi012 ceph-mon[27236]: pgmap v29: 97 pgs: 97 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.3 MiB/s wr, 526 op/s 2024-04-14T13:40:27.665 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:27 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:27.665 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:27 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:27.729 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:27 smithi008 ceph-mon[24786]: pgmap v29: 97 pgs: 97 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.3 MiB/s wr, 526 op/s 2024-04-14T13:40:27.729 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:27 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:27.729 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:27 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:28.337 INFO:teuthology.orchestra.run.smithi008.stdout: 5 153417 12.42 MB/sec execute 182 sec latency 41.831 ms 2024-04-14T13:40:28.753 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:28 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:28.753 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:28 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:28.754 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:28 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:40:28.754 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:28 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:40:28.754 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:28 smithi008 ceph-mon[24786]: pgmap v30: 97 pgs: 97 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 825 op/s 2024-04-14T13:40:28.754 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:28 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:28.754 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:28 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:28.754 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:28 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:40:28.754 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:28 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:40:28.754 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:28 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:28.754 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:28 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi008.yemlhr"}]: dispatch 2024-04-14T13:40:28.754 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:28 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi008.yemlhr"}]': finished 2024-04-14T13:40:28.754 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:28 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi012.ksvnxm"}]: dispatch 2024-04-14T13:40:28.754 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:28 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi012.ksvnxm"}]': finished 2024-04-14T13:40:28.754 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:28 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T13:40:29.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:28 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:29.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:28 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:29.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:28 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:40:29.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:28 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:40:29.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:28 smithi012 ceph-mon[27236]: pgmap v30: 97 pgs: 97 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 825 op/s 2024-04-14T13:40:29.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:28 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:29.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:28 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:29.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:28 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:40:29.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:28 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:40:29.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:28 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:29.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:28 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi008.yemlhr"}]: dispatch 2024-04-14T13:40:29.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:28 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi008.yemlhr"}]': finished 2024-04-14T13:40:29.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:28 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi012.ksvnxm"}]: dispatch 2024-04-14T13:40:29.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:28 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi012.ksvnxm"}]': finished 2024-04-14T13:40:29.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:28 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T13:40:29.337 INFO:teuthology.orchestra.run.smithi008.stdout: 5 153721 12.39 MB/sec execute 183 sec latency 30.148 ms 2024-04-14T13:40:29.761 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:29 smithi008 ceph-mon[24786]: Upgrade: Setting container_image for all mgr 2024-04-14T13:40:30.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:29 smithi012 ceph-mon[27236]: Upgrade: Setting container_image for all mgr 2024-04-14T13:40:30.337 INFO:teuthology.orchestra.run.smithi008.stdout: 5 154195 12.39 MB/sec execute 184 sec latency 35.358 ms 2024-04-14T13:40:30.737 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:30 smithi008 ceph-mon[24786]: pgmap v31: 97 pgs: 97 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.5 MiB/s wr, 525 op/s 2024-04-14T13:40:31.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:30 smithi012 ceph-mon[27236]: pgmap v31: 97 pgs: 97 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.5 MiB/s wr, 525 op/s 2024-04-14T13:40:31.337 INFO:teuthology.orchestra.run.smithi008.stdout: 5 154741 12.40 MB/sec execute 185 sec latency 32.960 ms 2024-04-14T13:40:32.026 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:31 smithi008 ceph-mon[24786]: Upgrade: Updating mon.smithi008 2024-04-14T13:40:32.026 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:31 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:32.026 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:31 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:40:32.026 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:31 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:40:32.026 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:31 smithi008 ceph-mon[24786]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:40:32.026 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:31 smithi008 ceph-mon[24786]: Deploying daemon mon.smithi008 on smithi008 2024-04-14T13:40:32.026 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:31 smithi008 ceph-mon[24786]: pgmap v32: 97 pgs: 97 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 813 op/s 2024-04-14T13:40:32.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:31 smithi012 ceph-mon[27236]: Upgrade: Updating mon.smithi008 2024-04-14T13:40:32.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:31 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:32.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:31 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:40:32.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:31 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:40:32.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:31 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:40:32.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:31 smithi012 ceph-mon[27236]: Deploying daemon mon.smithi008 on smithi008 2024-04-14T13:40:32.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:31 smithi012 ceph-mon[27236]: pgmap v32: 97 pgs: 97 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 813 op/s 2024-04-14T13:40:32.337 INFO:teuthology.orchestra.run.smithi008.stdout: 5 155350 12.42 MB/sec execute 186 sec latency 33.205 ms 2024-04-14T13:40:33.337 INFO:teuthology.orchestra.run.smithi008.stdout: 5 155805 12.41 MB/sec execute 187 sec latency 38.999 ms 2024-04-14T13:40:34.050 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:33 smithi008 ceph-mon[24786]: pgmap v33: 97 pgs: 97 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.7 MiB/s wr, 574 op/s 2024-04-14T13:40:34.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:33 smithi012 ceph-mon[27236]: pgmap v33: 97 pgs: 97 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.7 MiB/s wr, 574 op/s 2024-04-14T13:40:34.303 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:34 smithi008 systemd[1]: Stopping Ceph mon.smithi008 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:40:34.337 INFO:teuthology.orchestra.run.smithi008.stdout: 5 156079 12.41 MB/sec execute 188 sec latency 49.205 ms 2024-04-14T13:40:34.558 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:34 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008[24782]: 2024-04-14T13:40:34.453+0000 7f4c26c77700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi008 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:40:34.559 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:34 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008[24782]: 2024-04-14T13:40:34.453+0000 7f4c26c77700 -1 mon.smithi008@0(leader) e2 *** Got Signal Terminated *** 2024-04-14T13:40:34.559 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:34 smithi008 podman[66880]: 2024-04-14 13:40:34.496517502 +0000 UTC m=+0.230459290 container died 7786c27a60d0a0461d3eaa074797b473b59a422a1712ad98961c582e7de3898e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vendor=Red Hat, Inc., ceph=True, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, io.openshift.tags=base centos centos-stream, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, name=centos-stream, GIT_CLEAN=True) 2024-04-14T13:40:35.337 INFO:teuthology.orchestra.run.smithi008.stdout: 5 156476 12.41 MB/sec execute 189 sec latency 37.441 ms 2024-04-14T13:40:35.567 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:35 smithi008 podman[66880]: 2024-04-14 13:40:35.331620053 +0000 UTC m=+1.065561840 container cleanup 7786c27a60d0a0461d3eaa074797b473b59a422a1712ad98961c582e7de3898e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, vcs-type=git, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, name=centos-stream, GIT_BRANCH=HEAD, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754) 2024-04-14T13:40:35.567 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:35 smithi008 bash[66880]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008 2024-04-14T13:40:35.828 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:35 smithi008 podman[66893]: 2024-04-14 13:40:35.763972486 +0000 UTC m=+1.270363166 container remove 7786c27a60d0a0461d3eaa074797b473b59a422a1712ad98961c582e7de3898e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., ceph=True, build-date=2022-05-03T08:36:31.336870, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, GIT_BRANCH=HEAD, distribution-scope=public, RELEASE=HEAD, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8) 2024-04-14T13:40:36.215 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:36 smithi008 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@mon.smithi008.service: Deactivated successfully. 2024-04-14T13:40:36.215 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:36 smithi008 systemd[1]: Stopped Ceph mon.smithi008 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:40:36.215 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:36 smithi008 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@mon.smithi008.service: Consumed 17.070s CPU time. 2024-04-14T13:40:36.337 INFO:teuthology.orchestra.run.smithi008.stdout: 5 156955 12.41 MB/sec execute 190 sec latency 41.604 ms 2024-04-14T13:40:36.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:36 smithi008 systemd[1]: Starting Ceph mon.smithi008 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:40:36.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:36 smithi008 podman[67051]: 2024-04-14 13:40:36.657844418 +0000 UTC m=+0.025530721 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:40:36.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:36 smithi008 podman[67051]: 2024-04-14 13:40:36.806828772 +0000 UTC m=+0.174515152 container create 673dd6f4c8d803e16c0354275d3ad8ee2ca56b3d1f2920791f3265d81b95724e (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T13:40:37.221 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 podman[67051]: 2024-04-14 13:40:37.124431304 +0000 UTC m=+0.492117590 container init 673dd6f4c8d803e16c0354275d3ad8ee2ca56b3d1f2920791f3265d81b95724e (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS) 2024-04-14T13:40:37.221 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 podman[67051]: 2024-04-14 13:40:37.129790871 +0000 UTC m=+0.497477158 container start 673dd6f4c8d803e16c0354275d3ad8ee2ca56b3d1f2920791f3265d81b95724e (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-14T13:40:37.221 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: set uid:gid to 167:167 (ceph:ceph) 2024-04-14T13:40:37.221 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev), process ceph-mon, pid 2 2024-04-14T13:40:37.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: pidfile_write: ignore empty --pid-file 2024-04-14T13:40:37.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: load: jerasure load: lrc 2024-04-14T13:40:37.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: RocksDB version: 7.9.2 2024-04-14T13:40:37.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Git sha 0 2024-04-14T13:40:37.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Compile date 2024-04-06 20:28:24 2024-04-14T13:40:37.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: DB SUMMARY 2024-04-14T13:40:37.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: DB Session ID: DBSQE1NP64H7IUGROUN7 2024-04-14T13:40:37.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: CURRENT file: CURRENT 2024-04-14T13:40:37.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: IDENTITY file: IDENTITY 2024-04-14T13:40:37.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: MANIFEST file: MANIFEST-000015 size: 978 Bytes 2024-04-14T13:40:37.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi008/store.db dir, Total Num: 1, files: 000030.sst 2024-04-14T13:40:37.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi008/store.db: 000028.log size: 2882892 ; 2024-04-14T13:40:37.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.error_if_exists: 0 2024-04-14T13:40:37.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.create_if_missing: 0 2024-04-14T13:40:37.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.paranoid_checks: 1 2024-04-14T13:40:37.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-14T13:40:37.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-14T13:40:37.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-14T13:40:37.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.env: 0x55e394424c60 2024-04-14T13:40:37.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.fs: PosixFileSystem 2024-04-14T13:40:37.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.info_log: 0x55e396541aa0 2024-04-14T13:40:37.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_file_opening_threads: 16 2024-04-14T13:40:37.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.statistics: (nil) 2024-04-14T13:40:37.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.use_fsync: 0 2024-04-14T13:40:37.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_log_file_size: 0 2024-04-14T13:40:37.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-14T13:40:37.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-14T13:40:37.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.keep_log_file_num: 1000 2024-04-14T13:40:37.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.recycle_log_file_num: 0 2024-04-14T13:40:37.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.allow_fallocate: 1 2024-04-14T13:40:37.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.allow_mmap_reads: 0 2024-04-14T13:40:37.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.allow_mmap_writes: 0 2024-04-14T13:40:37.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.use_direct_reads: 0 2024-04-14T13:40:37.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-14T13:40:37.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.create_missing_column_families: 0 2024-04-14T13:40:37.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.db_log_dir: 2024-04-14T13:40:37.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.wal_dir: 2024-04-14T13:40:37.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-14T13:40:37.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-14T13:40:37.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-14T13:40:37.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-14T13:40:37.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-14T13:40:37.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-14T13:40:37.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.advise_random_on_open: 1 2024-04-14T13:40:37.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.db_write_buffer_size: 0 2024-04-14T13:40:37.225 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.write_buffer_manager: 0x55e3965121e0 2024-04-14T13:40:37.225 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-14T13:40:37.225 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-14T13:40:37.225 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-14T13:40:37.225 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.rate_limiter: (nil) 2024-04-14T13:40:37.225 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-14T13:40:37.225 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.wal_recovery_mode: 2 2024-04-14T13:40:37.225 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.enable_thread_tracking: 0 2024-04-14T13:40:37.225 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.enable_pipelined_write: 0 2024-04-14T13:40:37.225 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.unordered_write: 0 2024-04-14T13:40:37.225 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-14T13:40:37.225 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-14T13:40:37.225 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-14T13:40:37.225 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-14T13:40:37.225 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.row_cache: None 2024-04-14T13:40:37.225 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.wal_filter: None 2024-04-14T13:40:37.226 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-14T13:40:37.226 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.allow_ingest_behind: 0 2024-04-14T13:40:37.226 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.two_write_queues: 0 2024-04-14T13:40:37.226 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.manual_wal_flush: 0 2024-04-14T13:40:37.227 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.wal_compression: 0 2024-04-14T13:40:37.227 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.atomic_flush: 0 2024-04-14T13:40:37.227 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-14T13:40:37.227 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-14T13:40:37.227 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-14T13:40:37.227 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.log_readahead_size: 0 2024-04-14T13:40:37.227 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-14T13:40:37.227 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.best_efforts_recovery: 0 2024-04-14T13:40:37.227 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-14T13:40:37.227 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-14T13:40:37.227 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.allow_data_in_errors: 0 2024-04-14T13:40:37.227 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.db_host_id: __hostname__ 2024-04-14T13:40:37.227 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-14T13:40:37.227 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_background_jobs: 2 2024-04-14T13:40:37.227 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_background_compactions: -1 2024-04-14T13:40:37.227 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_subcompactions: 1 2024-04-14T13:40:37.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-14T13:40:37.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-14T13:40:37.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-14T13:40:37.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_total_wal_size: 0 2024-04-14T13:40:37.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-14T13:40:37.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-14T13:40:37.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-14T13:40:37.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-14T13:40:37.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_open_files: -1 2024-04-14T13:40:37.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.bytes_per_sync: 0 2024-04-14T13:40:37.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-14T13:40:37.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-14T13:40:37.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compaction_readahead_size: 0 2024-04-14T13:40:37.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_background_flushes: -1 2024-04-14T13:40:37.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Compression algorithms supported: 2024-04-14T13:40:37.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-14T13:40:37.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: kZSTD supported: 0 2024-04-14T13:40:37.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: kXpressCompression supported: 0 2024-04-14T13:40:37.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: kLZ4HCCompression supported: 1 2024-04-14T13:40:37.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: kZlibCompression supported: 1 2024-04-14T13:40:37.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: kSnappyCompression supported: 1 2024-04-14T13:40:37.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: kLZ4Compression supported: 1 2024-04-14T13:40:37.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: kBZip2Compression supported: 0 2024-04-14T13:40:37.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-14T13:40:37.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-14T13:40:37.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi008/store.db/MANIFEST-000015 2024-04-14T13:40:37.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-14T13:40:37.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-14T13:40:37.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.merge_operator: 2024-04-14T13:40:37.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compaction_filter: None 2024-04-14T13:40:37.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compaction_filter_factory: None 2024-04-14T13:40:37.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.sst_partitioner_factory: None 2024-04-14T13:40:37.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-14T13:40:37.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-14T13:40:37.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e396541c80) 2024-04-14T13:40:37.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: cache_index_and_filter_blocks: 1 2024-04-14T13:40:37.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-14T13:40:37.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-14T13:40:37.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: pin_top_level_index_and_filter: 1 2024-04-14T13:40:37.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: index_type: 0 2024-04-14T13:40:37.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: data_block_index_type: 0 2024-04-14T13:40:37.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: index_shortening: 1 2024-04-14T13:40:37.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-14T13:40:37.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: checksum: 4 2024-04-14T13:40:37.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: no_block_cache: 0 2024-04-14T13:40:37.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache: 0x55e396545090 2024-04-14T13:40:37.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache_name: BinnedLRUCache 2024-04-14T13:40:37.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache_options: 2024-04-14T13:40:37.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: capacity : 536870912 2024-04-14T13:40:37.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: num_shard_bits : 4 2024-04-14T13:40:37.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: strict_capacity_limit : 0 2024-04-14T13:40:37.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: high_pri_pool_ratio: 0.000 2024-04-14T13:40:37.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache_compressed: (nil) 2024-04-14T13:40:37.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: persistent_cache: (nil) 2024-04-14T13:40:37.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_size: 4096 2024-04-14T13:40:37.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_size_deviation: 10 2024-04-14T13:40:37.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_restart_interval: 16 2024-04-14T13:40:37.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: index_block_restart_interval: 1 2024-04-14T13:40:37.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: metadata_block_size: 4096 2024-04-14T13:40:37.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: partition_filters: 0 2024-04-14T13:40:37.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: use_delta_encoding: 1 2024-04-14T13:40:37.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: filter_policy: bloomfilter 2024-04-14T13:40:37.232 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: whole_key_filtering: 1 2024-04-14T13:40:37.232 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: verify_compression: 0 2024-04-14T13:40:37.232 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: read_amp_bytes_per_bit: 0 2024-04-14T13:40:37.232 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: format_version: 5 2024-04-14T13:40:37.232 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: enable_index_compression: 1 2024-04-14T13:40:37.232 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_align: 0 2024-04-14T13:40:37.232 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: max_auto_readahead_size: 262144 2024-04-14T13:40:37.232 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: prepopulate_block_cache: 0 2024-04-14T13:40:37.232 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: initial_auto_readahead_size: 8192 2024-04-14T13:40:37.232 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: num_file_reads_for_auto_readahead: 2 2024-04-14T13:40:37.232 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.write_buffer_size: 33554432 2024-04-14T13:40:37.232 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_write_buffer_number: 2 2024-04-14T13:40:37.232 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compression: NoCompression 2024-04-14T13:40:37.232 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.bottommost_compression: Disabled 2024-04-14T13:40:37.232 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.prefix_extractor: nullptr 2024-04-14T13:40:37.232 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-14T13:40:37.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.num_levels: 7 2024-04-14T13:40:37.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-14T13:40:37.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-14T13:40:37.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-14T13:40:37.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-14T13:40:37.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-14T13:40:37.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-14T13:40:37.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-14T13:40:37.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-14T13:40:37.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-14T13:40:37.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-14T13:40:37.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-14T13:40:37.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-14T13:40:37.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-14T13:40:37.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compression_opts.level: 32767 2024-04-14T13:40:37.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compression_opts.strategy: 0 2024-04-14T13:40:37.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-14T13:40:37.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-14T13:40:37.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-14T13:40:37.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-14T13:40:37.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compression_opts.enabled: false 2024-04-14T13:40:37.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-14T13:40:37.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-14T13:40:37.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-14T13:40:37.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-14T13:40:37.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.target_file_size_base: 67108864 2024-04-14T13:40:37.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-14T13:40:37.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-14T13:40:37.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-14T13:40:37.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-14T13:40:37.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-14T13:40:37.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-14T13:40:37.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-14T13:40:37.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-14T13:40:37.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-14T13:40:37.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-14T13:40:37.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-14T13:40:37.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-14T13:40:37.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-14T13:40:37.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-14T13:40:37.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.arena_block_size: 1048576 2024-04-14T13:40:37.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-14T13:40:37.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-14T13:40:37.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.disable_auto_compactions: 0 2024-04-14T13:40:37.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-14T13:40:37.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-14T13:40:37.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-14T13:40:37.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-14T13:40:37.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-14T13:40:37.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-14T13:40:37.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-14T13:40:37.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-14T13:40:37.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-14T13:40:37.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-14T13:40:37.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-14T13:40:37.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.inplace_update_support: 0 2024-04-14T13:40:37.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-14T13:40:37.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-14T13:40:37.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-14T13:40:37.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-14T13:40:37.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.bloom_locality: 0 2024-04-14T13:40:37.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.max_successive_merges: 0 2024-04-14T13:40:37.237 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-14T13:40:37.237 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.paranoid_file_checks: 0 2024-04-14T13:40:37.237 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.force_consistency_checks: 1 2024-04-14T13:40:37.237 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.report_bg_io_stats: 0 2024-04-14T13:40:37.237 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.ttl: 2592000 2024-04-14T13:40:37.237 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-14T13:40:37.237 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-14T13:40:37.238 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-14T13:40:37.238 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.enable_blob_files: false 2024-04-14T13:40:37.238 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.min_blob_size: 0 2024-04-14T13:40:37.238 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.blob_file_size: 268435456 2024-04-14T13:40:37.238 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-14T13:40:37.238 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-14T13:40:37.238 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-14T13:40:37.238 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-14T13:40:37.238 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-14T13:40:37.238 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.blob_file_starting_level: 0 2024-04-14T13:40:37.238 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-14T13:40:37.238 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-04-14T13:40:37.238 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-14T13:40:37.239 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi008/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 11835, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-14T13:40:37.239 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-04-14T13:40:37.239 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8baabb18-a9dc-4b86-9195-2a945687bc19 2024-04-14T13:40:37.239 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713102037171093, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-04-14T13:40:37.239 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-04-14T13:40:37.239 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713102037183408, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2662969, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11836, "largest_seqno": 13063, "table_properties": {"data_size": 2657005, "index_size": 3575, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1349, "raw_key_size": 11825, "raw_average_key_size": 22, "raw_value_size": 2645793, "raw_average_value_size": 5117, "num_data_blocks": 162, "num_entries": 517, "num_filter_entries": 517, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713102037, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8baabb18-a9dc-4b86-9195-2a945687bc19", "db_session_id": "DBSQE1NP64H7IUGROUN7", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-04-14T13:40:37.239 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713102037183624, "job": 1, "event": "recovery_finished"} 2024-04-14T13:40:37.239 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-04-14T13:40:37.239 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-14T13:40:37.239 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi008/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-14T13:40:37.239 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e396634000 2024-04-14T13:40:37.239 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: DB pointer 0x55e39661e000 2024-04-14T13:40:37.239 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-14T13:40:37.239 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-14T13:40:37.239 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** DB Stats ** 2024-04-14T13:40:37.239 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T13:40:37.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-14T13:40:37.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T13:40:37.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T13:40:37.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-14T13:40:37.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T13:40:37.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T13:40:37.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-14T13:40:37.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** Compaction Stats [default] ** 2024-04-14T13:40:37.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T13:40:37.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-14T13:40:37.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: L0 1/0 2.54 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 396.5 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-14T13:40:37.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: L6 1/0 6.52 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-14T13:40:37.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Sum 2/0 9.06 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 396.5 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-14T13:40:37.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 396.5 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-14T13:40:37.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-14T13:40:37.241 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** Compaction Stats [default] ** 2024-04-14T13:40:37.241 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T13:40:37.241 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T13:40:37.241 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 396.5 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-14T13:40:37.241 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-14T13:40:37.241 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-14T13:40:37.241 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-14T13:40:37.241 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T13:40:37.241 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-04-14T13:40:37.241 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T13:40:37.241 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T13:40:37.241 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T13:40:37.241 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T13:40:37.241 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative compaction: 0.00 GB write, 152.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T13:40:37.241 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval compaction: 0.00 GB write, 152.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T13:40:37.241 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T13:40:37.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Block cache BinnedLRUCache@0x55e396545090#2 capacity: 512.00 MB usage: 5.48 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-04-14T13:40:37.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.41 KB,0.000268221%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-04-14T13:40:37.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-14T13:40:37.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T13:40:37.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: starting mon.smithi008 rank 0 at public addrs [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] at bind addrs [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi008 fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:40:37.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: mon.smithi008@-1(???) e2 preinit fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:40:37.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: mon.smithi008@-1(???).mds e7 new map 2024-04-14T13:40:37.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: mon.smithi008@-1(???).mds e7 print_map 2024-04-14T13:40:37.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: e7 2024-04-14T13:40:37.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-14T13:40:37.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-14T13:40:37.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: legacy client fscid: 1 2024-04-14T13:40:37.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-14T13:40:37.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Filesystem 'foofs' (1) 2024-04-14T13:40:37.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: fs_name foofs 2024-04-14T13:40:37.243 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: epoch 5 2024-04-14T13:40:37.243 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-14T13:40:37.243 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: created 2024-04-14T13:34:52.552946+0000 2024-04-14T13:40:37.243 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: modified 2024-04-14T13:34:58.569998+0000 2024-04-14T13:40:37.243 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: tableserver 0 2024-04-14T13:40:37.243 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: root 0 2024-04-14T13:40:37.243 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: session_timeout 60 2024-04-14T13:40:37.243 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: session_autoclose 300 2024-04-14T13:40:37.243 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: max_file_size 1099511627776 2024-04-14T13:40:37.243 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: max_xattr_size 65536 2024-04-14T13:40:37.243 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: required_client_features {} 2024-04-14T13:40:37.244 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: last_failure 0 2024-04-14T13:40:37.244 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: last_failure_osd_epoch 0 2024-04-14T13:40:37.244 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-14T13:40:37.244 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: max_mds 1 2024-04-14T13:40:37.244 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: in 0 2024-04-14T13:40:37.244 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: up {0=14506} 2024-04-14T13:40:37.244 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: failed 2024-04-14T13:40:37.244 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: damaged 2024-04-14T13:40:37.244 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: stopped 2024-04-14T13:40:37.244 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: data_pools [3] 2024-04-14T13:40:37.244 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: metadata_pool 2 2024-04-14T13:40:37.244 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: inline_data disabled 2024-04-14T13:40:37.244 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: balancer 2024-04-14T13:40:37.244 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: bal_rank_mask -1 2024-04-14T13:40:37.244 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: standby_count_wanted 1 2024-04-14T13:40:37.244 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: qdb_cluster leader: 0 members: 2024-04-14T13:40:37.245 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: [mds.foofs.smithi008.wqbvfi{0:14506} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.8:6834/785972446,v1:172.21.15.8:6835/785972446] compat {c=[1],r=[1],i=[7ff]}] 2024-04-14T13:40:37.245 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-14T13:40:37.245 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-14T13:40:37.245 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Standby daemons: 2024-04-14T13:40:37.245 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-14T13:40:37.245 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: [mds.foofs.smithi012.vjovzp{-1:24309} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.12:6832/1210020061,v1:172.21.15.12:6833/1210020061] compat {c=[1],r=[1],i=[7ff]}] 2024-04-14T13:40:37.245 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: mon.smithi008@-1(???).osd e72 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-14T13:40:37.245 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: mon.smithi008@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T13:40:37.245 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: mon.smithi008@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T13:40:37.245 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: mon.smithi008@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T13:40:37.245 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: mon.smithi008@-1(???).paxosservice(auth 1..29) refresh upgraded, format 0 -> 3 2024-04-14T13:40:37.338 INFO:teuthology.orchestra.run.smithi008.stdout: 5 157377 12.41 MB/sec execute 191 sec latency 30.327 ms 2024-04-14T13:40:37.471 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 bash[67051]: 673dd6f4c8d803e16c0354275d3ad8ee2ca56b3d1f2920791f3265d81b95724e 2024-04-14T13:40:37.471 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 systemd[1]: Started Ceph mon.smithi008 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:40:37.977 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:40:37.977 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:40:37.977 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:40:37.977 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: pgmap v34: 97 pgs: 97 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.7 MiB/s wr, 574 op/s 2024-04-14T13:40:37.977 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: pgmap v35: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 839 op/s 2024-04-14T13:40:37.977 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: mon.smithi008 calling monitor election 2024-04-14T13:40:37.977 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: mon.smithi008 is new leader, mons smithi008,smithi012 in quorum (ranks 0,1) 2024-04-14T13:40:37.977 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: monmap epoch 2 2024-04-14T13:40:37.977 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:40:37.977 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: last_changed 2024-04-14T13:29:52.242401+0000 2024-04-14T13:40:37.978 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: created 2024-04-14T13:27:27.599436+0000 2024-04-14T13:40:37.978 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: min_mon_release 17 (quincy) 2024-04-14T13:40:37.978 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: election_strategy: 1 2024-04-14T13:40:37.978 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: 0: [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon.smithi008 2024-04-14T13:40:37.978 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: 1: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.smithi012 2024-04-14T13:40:37.978 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: fsmap foofs:1 {0=foofs.smithi008.wqbvfi=up:active} 1 up:standby 2024-04-14T13:40:37.978 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: osdmap e72: 8 total, 8 up, 8 in 2024-04-14T13:40:37.978 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: mgrmap e34: smithi008.yemlhr(active, since 63s), standbys: smithi012.ksvnxm 2024-04-14T13:40:37.978 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T13:40:37.978 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:40:37.978 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: daemon grafana.smithi008 on smithi008 is in unknown state 2024-04-14T13:40:37.978 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: mgrmap e35: smithi008.yemlhr(active, since 63s), standbys: smithi012.ksvnxm 2024-04-14T13:40:37.978 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:37 smithi008 ceph-mon[67070]: from='mgr.14794 ' entity='' 2024-04-14T13:40:38.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:40:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:40:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: from='mgr.14794 172.21.15.8:0/2304025329' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:40:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: pgmap v34: 97 pgs: 97 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.7 MiB/s wr, 574 op/s 2024-04-14T13:40:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: pgmap v35: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 839 op/s 2024-04-14T13:40:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: mon.smithi008 calling monitor election 2024-04-14T13:40:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: mon.smithi008 is new leader, mons smithi008,smithi012 in quorum (ranks 0,1) 2024-04-14T13:40:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: monmap epoch 2 2024-04-14T13:40:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:40:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: last_changed 2024-04-14T13:29:52.242401+0000 2024-04-14T13:40:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: created 2024-04-14T13:27:27.599436+0000 2024-04-14T13:40:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: min_mon_release 17 (quincy) 2024-04-14T13:40:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: election_strategy: 1 2024-04-14T13:40:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: 0: [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon.smithi008 2024-04-14T13:40:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: 1: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.smithi012 2024-04-14T13:40:38.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: fsmap foofs:1 {0=foofs.smithi008.wqbvfi=up:active} 1 up:standby 2024-04-14T13:40:38.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: osdmap e72: 8 total, 8 up, 8 in 2024-04-14T13:40:38.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: mgrmap e34: smithi008.yemlhr(active, since 63s), standbys: smithi012.ksvnxm 2024-04-14T13:40:38.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T13:40:38.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:40:38.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: daemon grafana.smithi008 on smithi008 is in unknown state 2024-04-14T13:40:38.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: mgrmap e35: smithi008.yemlhr(active, since 63s), standbys: smithi012.ksvnxm 2024-04-14T13:40:38.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:37 smithi012 ceph-mon[27236]: from='mgr.14794 ' entity='' 2024-04-14T13:40:38.339 INFO:teuthology.orchestra.run.smithi008.stdout: 5 157902 12.38 MB/sec execute 192 sec latency 29.752 ms 2024-04-14T13:40:39.339 INFO:teuthology.orchestra.run.smithi008.stdout: 5 158388 12.39 MB/sec execute 193 sec latency 31.156 ms 2024-04-14T13:40:40.019 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:39 smithi008 ceph-mon[67070]: mgrmap e36: smithi008.yemlhr(active, since 64s), standbys: smithi012.ksvnxm 2024-04-14T13:40:40.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:39 smithi012 ceph-mon[27236]: mgrmap e36: smithi008.yemlhr(active, since 64s), standbys: smithi012.ksvnxm 2024-04-14T13:40:40.339 INFO:teuthology.orchestra.run.smithi008.stdout: 5 158921 12.40 MB/sec execute 194 sec latency 31.178 ms 2024-04-14T13:40:41.339 INFO:teuthology.orchestra.run.smithi008.stdout: 5 159370 12.40 MB/sec execute 195 sec latency 36.905 ms 2024-04-14T13:40:42.339 INFO:teuthology.orchestra.run.smithi008.stdout: 5 159773 12.43 MB/sec execute 196 sec latency 36.317 ms 2024-04-14T13:40:43.339 INFO:teuthology.orchestra.run.smithi008.stdout: 5 160216 12.42 MB/sec execute 197 sec latency 35.890 ms 2024-04-14T13:40:44.339 INFO:teuthology.orchestra.run.smithi008.stdout: 5 160540 12.39 MB/sec execute 198 sec latency 39.687 ms 2024-04-14T13:40:45.339 INFO:teuthology.orchestra.run.smithi008.stdout: 5 160752 12.35 MB/sec execute 199 sec latency 43.051 ms 2024-04-14T13:40:46.339 INFO:teuthology.orchestra.run.smithi008.stdout: 5 161049 12.33 MB/sec execute 200 sec latency 50.036 ms 2024-04-14T13:40:46.357 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:46 smithi008 ceph-mon[67070]: Standby manager daemon smithi012.ksvnxm restarted 2024-04-14T13:40:46.357 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:46 smithi008 ceph-mon[67070]: Standby manager daemon smithi012.ksvnxm started 2024-04-14T13:40:46.357 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:46 smithi008 ceph-mon[67070]: from='mgr.? 172.21.15.12:0/4166967949' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi012.ksvnxm/crt"}]: dispatch 2024-04-14T13:40:46.357 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:46 smithi008 ceph-mon[67070]: from='mgr.? 172.21.15.12:0/4166967949' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T13:40:46.357 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:46 smithi008 ceph-mon[67070]: from='mgr.? 172.21.15.12:0/4166967949' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi012.ksvnxm/key"}]: dispatch 2024-04-14T13:40:46.358 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:46 smithi008 ceph-mon[67070]: from='mgr.? 172.21.15.12:0/4166967949' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T13:40:46.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:46 smithi012 ceph-mon[27236]: Standby manager daemon smithi012.ksvnxm restarted 2024-04-14T13:40:46.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:46 smithi012 ceph-mon[27236]: Standby manager daemon smithi012.ksvnxm started 2024-04-14T13:40:46.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:46 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.12:0/4166967949' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi012.ksvnxm/crt"}]: dispatch 2024-04-14T13:40:46.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:46 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.12:0/4166967949' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T13:40:46.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:46 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.12:0/4166967949' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi012.ksvnxm/key"}]: dispatch 2024-04-14T13:40:46.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:46 smithi012 ceph-mon[27236]: from='mgr.? 172.21.15.12:0/4166967949' entity='mgr.smithi012.ksvnxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T13:40:47.339 INFO:teuthology.orchestra.run.smithi008.stdout: 5 161375 12.30 MB/sec execute 201 sec latency 68.671 ms 2024-04-14T13:40:47.386 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:47 smithi008 ceph-mon[67070]: mgrmap e37: smithi008.yemlhr(active, since 71s), standbys: smithi012.ksvnxm 2024-04-14T13:40:47.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:47 smithi012 ceph-mon[27236]: mgrmap e37: smithi008.yemlhr(active, since 71s), standbys: smithi012.ksvnxm 2024-04-14T13:40:48.339 INFO:teuthology.orchestra.run.smithi008.stdout: 5 161658 12.27 MB/sec execute 202 sec latency 56.777 ms 2024-04-14T13:40:48.417 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: Active manager daemon smithi008.yemlhr restarted 2024-04-14T13:40:48.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: Activating manager daemon smithi008.yemlhr 2024-04-14T13:40:48.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: osdmap e73: 8 total, 8 up, 8 in 2024-04-14T13:40:48.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: mgrmap e38: smithi008.yemlhr(active, starting, since 0.0187666s), standbys: smithi012.ksvnxm 2024-04-14T13:40:48.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:40:48.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:40:48.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wqbvfi"}]: dispatch 2024-04-14T13:40:48.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi012.vjovzp"}]: dispatch 2024-04-14T13:40:48.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr metadata", "who": "smithi008.yemlhr", "id": "smithi008.yemlhr"}]: dispatch 2024-04-14T13:40:48.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr metadata", "who": "smithi012.ksvnxm", "id": "smithi012.ksvnxm"}]: dispatch 2024-04-14T13:40:48.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:40:48.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:40:48.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:40:48.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:40:48.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:40:48.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:40:48.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:40:48.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:40:48.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T13:40:48.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T13:40:48.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T13:40:48.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: Manager daemon smithi008.yemlhr is now available 2024-04-14T13:40:48.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:40:48.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:40:48.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:40:48.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:40:48.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:40:48.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:48 smithi008 ceph-mon[67070]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:40:48.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: Active manager daemon smithi008.yemlhr restarted 2024-04-14T13:40:48.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: Activating manager daemon smithi008.yemlhr 2024-04-14T13:40:48.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: osdmap e73: 8 total, 8 up, 8 in 2024-04-14T13:40:48.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: mgrmap e38: smithi008.yemlhr(active, starting, since 0.0187666s), standbys: smithi012.ksvnxm 2024-04-14T13:40:48.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:40:48.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:40:48.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wqbvfi"}]: dispatch 2024-04-14T13:40:48.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi012.vjovzp"}]: dispatch 2024-04-14T13:40:48.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr metadata", "who": "smithi008.yemlhr", "id": "smithi008.yemlhr"}]: dispatch 2024-04-14T13:40:48.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr metadata", "who": "smithi012.ksvnxm", "id": "smithi012.ksvnxm"}]: dispatch 2024-04-14T13:40:48.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:40:48.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:40:48.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:40:48.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:40:48.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:40:48.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:40:48.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:40:48.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:40:48.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T13:40:48.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T13:40:48.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T13:40:48.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: Manager daemon smithi008.yemlhr is now available 2024-04-14T13:40:48.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:40:48.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:40:48.803 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:40:48.803 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:40:48.803 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:40:48.803 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:48 smithi012 ceph-mon[27236]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.yemlhr/trash_purge_schedule"}]: dispatch 2024-04-14T13:40:49.340 INFO:teuthology.orchestra.run.smithi008.stdout: 5 161855 12.24 MB/sec execute 203 sec latency 43.229 ms 2024-04-14T13:40:50.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:50 smithi008 ceph-mon[67070]: mgrmap e39: smithi008.yemlhr(active, since 1.03017s), standbys: smithi012.ksvnxm 2024-04-14T13:40:50.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:50 smithi008 ceph-mon[67070]: [14/Apr/2024:13:40:49] ENGINE Bus STARTING 2024-04-14T13:40:50.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:50 smithi008 ceph-mon[67070]: [14/Apr/2024:13:40:49] ENGINE Serving on http://172.21.15.8:8765 2024-04-14T13:40:50.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:50 smithi008 ceph-mon[67070]: [14/Apr/2024:13:40:49] ENGINE Serving on https://172.21.15.8:7150 2024-04-14T13:40:50.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:50 smithi008 ceph-mon[67070]: [14/Apr/2024:13:40:49] ENGINE Bus STARTED 2024-04-14T13:40:50.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:50 smithi012 ceph-mon[27236]: mgrmap e39: smithi008.yemlhr(active, since 1.03017s), standbys: smithi012.ksvnxm 2024-04-14T13:40:50.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:50 smithi012 ceph-mon[27236]: [14/Apr/2024:13:40:49] ENGINE Bus STARTING 2024-04-14T13:40:50.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:50 smithi012 ceph-mon[27236]: [14/Apr/2024:13:40:49] ENGINE Serving on http://172.21.15.8:8765 2024-04-14T13:40:50.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:50 smithi012 ceph-mon[27236]: [14/Apr/2024:13:40:49] ENGINE Serving on https://172.21.15.8:7150 2024-04-14T13:40:50.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:50 smithi012 ceph-mon[27236]: [14/Apr/2024:13:40:49] ENGINE Bus STARTED 2024-04-14T13:40:50.339 INFO:teuthology.orchestra.run.smithi008.stdout: 5 162116 12.23 MB/sec execute 204 sec latency 79.607 ms 2024-04-14T13:40:51.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:51 smithi008 ceph-mon[67070]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T13:40:51.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:51 smithi008 ceph-mon[67070]: Cluster is now healthy 2024-04-14T13:40:51.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:51 smithi008 ceph-mon[67070]: mgrmap e40: smithi008.yemlhr(active, since 2s), standbys: smithi012.ksvnxm 2024-04-14T13:40:51.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:51 smithi008 ceph-mon[67070]: pgmap v4: 97 pgs: 97 active+clean; 326 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-04-14T13:40:51.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:51 smithi012 ceph-mon[27236]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T13:40:51.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:51 smithi012 ceph-mon[27236]: Cluster is now healthy 2024-04-14T13:40:51.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:51 smithi012 ceph-mon[27236]: mgrmap e40: smithi008.yemlhr(active, since 2s), standbys: smithi012.ksvnxm 2024-04-14T13:40:51.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:51 smithi012 ceph-mon[27236]: pgmap v4: 97 pgs: 97 active+clean; 326 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-04-14T13:40:51.340 INFO:teuthology.orchestra.run.smithi008.stdout: 5 162659 12.22 MB/sec execute 205 sec latency 58.338 ms 2024-04-14T13:40:52.340 INFO:teuthology.orchestra.run.smithi008.stdout: 5 163152 12.25 MB/sec execute 206 sec latency 37.141 ms 2024-04-14T13:40:53.249 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:53 smithi008 ceph-mon[67070]: pgmap v5: 97 pgs: 97 active+clean; 326 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-04-14T13:40:53.249 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:53 smithi008 ceph-mon[67070]: mgrmap e41: smithi008.yemlhr(active, since 4s), standbys: smithi012.ksvnxm 2024-04-14T13:40:53.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:53 smithi012 ceph-mon[27236]: pgmap v5: 97 pgs: 97 active+clean; 326 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-04-14T13:40:53.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:53 smithi012 ceph-mon[27236]: mgrmap e41: smithi008.yemlhr(active, since 4s), standbys: smithi012.ksvnxm 2024-04-14T13:40:53.339 INFO:teuthology.orchestra.run.smithi008.stdout: 5 163731 12.28 MB/sec execute 207 sec latency 30.877 ms 2024-04-14T13:40:54.251 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:54 smithi008 ceph-mon[67070]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:54.251 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:54 smithi008 ceph-mon[67070]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:54.339 INFO:teuthology.orchestra.run.smithi008.stdout: 5 164381 12.29 MB/sec execute 208 sec latency 37.462 ms 2024-04-14T13:40:54.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:54 smithi012 ceph-mon[27236]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:54.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:54 smithi012 ceph-mon[27236]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:55.340 INFO:teuthology.orchestra.run.smithi008.stdout: 5 165011 12.29 MB/sec execute 209 sec latency 30.286 ms 2024-04-14T13:40:55.770 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:55 smithi008 ceph-mon[67070]: pgmap v6: 97 pgs: 97 active+clean; 326 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-04-14T13:40:55.771 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:55 smithi008 ceph-mon[67070]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:55.771 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:55 smithi008 ceph-mon[67070]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:55.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:55 smithi012 ceph-mon[27236]: pgmap v6: 97 pgs: 97 active+clean; 326 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-04-14T13:40:55.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:55 smithi012 ceph-mon[27236]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:55.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:55 smithi012 ceph-mon[27236]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:56.121 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:40:56.340 INFO:teuthology.orchestra.run.smithi008.stdout: 5 165464 12.30 MB/sec execute 210 sec latency 57.675 ms 2024-04-14T13:40:57.153 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:40:57.153 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (2m) 29s ago 11m 16.8M - 0.23.0 ba2b418f427c cfddb9a7f858 2024-04-14T13:40:57.153 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (11m) 29s ago 11m 6405k - 17.2.0 e1d6a67b021e 68b810a9ff93 2024-04-14T13:40:57.153 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (11m) 3s ago 11m 6400k - 17.2.0 e1d6a67b021e 6bc31c51b938 2024-04-14T13:40:57.153 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 unknown 29s ago 11m - - 2024-04-14T13:40:57.153 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (6m) 29s ago 6m 37.7M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:40:57.153 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (5m) 3s ago 5m 13.8M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:40:57.153 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (101s) 29s ago 13m 497M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:40:57.153 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (41s) 3s ago 11m 448M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:40:57.153 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (13m) 29s ago 13m 60.3M 2048M 17.2.0 e1d6a67b021e 7786c27a60d0 2024-04-14T13:40:57.154 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (11m) 3s ago 11m 51.0M 2048M 17.2.0 e1d6a67b021e f6cd483dfa31 2024-04-14T13:40:57.154 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (5m) 29s ago 5m 275M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:40:57.154 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (5m) 3s ago 5m 87.9M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:40:57.154 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (2m) 29s ago 11m 10.4M - 1.3.1 1dbe0e931976 646dfca70be4 2024-04-14T13:40:57.154 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (113s) 3s ago 10m 18.2M - 1.3.1 1dbe0e931976 0a60dcd4044c 2024-04-14T13:40:57.154 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (10m) 29s ago 10m 519M 1937M 17.2.0 e1d6a67b021e 3b8cdea7890c 2024-04-14T13:40:57.154 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (9m) 29s ago 9m 452M 1937M 17.2.0 e1d6a67b021e be86642c7790 2024-04-14T13:40:57.154 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (9m) 29s ago 9m 315M 1937M 17.2.0 e1d6a67b021e b1017f00578f 2024-04-14T13:40:57.154 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (9m) 29s ago 9m 488M 1937M 17.2.0 e1d6a67b021e 66fca4b191be 2024-04-14T13:40:57.154 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (8m) 3s ago 8m 558M 2705M 17.2.0 e1d6a67b021e 43b02cb64951 2024-04-14T13:40:57.154 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (8m) 3s ago 8m 320M 2705M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:40:57.154 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (7m) 3s ago 7m 526M 2705M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:40:57.154 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (7m) 3s ago 7m 441M 2705M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:40:57.154 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (49s) 29s ago 10m 50.8M - 2.33.4 514e6a882f6e 12f97f75d620 2024-04-14T13:40:57.340 INFO:teuthology.orchestra.run.smithi008.stdout: 5 165848 12.29 MB/sec execute 211 sec latency 52.431 ms 2024-04-14T13:40:57.727 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:40:57.727 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:40:57.727 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-14T13:40:57.727 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T13:40:57.727 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:40:57.727 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:40:57.727 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:40:57.728 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:40:57.728 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:40:57.728 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T13:40:57.728 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:40:57.728 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:40:57.728 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:40:57.728 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:40:57.728 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:40:57.728 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-04-14T13:40:57.728 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T13:40:57.728 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:40:57.728 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:40:57.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:57 smithi012 ceph-mon[27236]: pgmap v7: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 34 KiB/s rd, 5.2 MiB/s wr, 497 op/s 2024-04-14T13:40:57.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:57 smithi012 ceph-mon[27236]: from='client.34128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:40:57.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:57 smithi012 ceph-mon[27236]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:40:57.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:57 smithi012 ceph-mon[27236]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:57.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:57 smithi012 ceph-mon[27236]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:57.809 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:57 smithi008 ceph-mon[67070]: pgmap v7: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 34 KiB/s rd, 5.2 MiB/s wr, 497 op/s 2024-04-14T13:40:57.809 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:57 smithi008 ceph-mon[67070]: from='client.34128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:40:57.809 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:57 smithi008 ceph-mon[67070]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:40:57.810 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:57 smithi008 ceph-mon[67070]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:57.810 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:57 smithi008 ceph-mon[67070]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:58.234 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:40:58.234 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:40:58.235 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:40:58.235 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:40:58.235 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-14T13:40:58.235 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-14T13:40:58.235 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:40:58.235 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "3/23 daemons upgraded", 2024-04-14T13:40:58.235 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "", 2024-04-14T13:40:58.235 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:40:58.235 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:40:58.340 INFO:teuthology.orchestra.run.smithi008.stdout: 5 166160 12.27 MB/sec execute 212 sec latency 63.107 ms 2024-04-14T13:40:58.792 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-04-14T13:40:58.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:58 smithi008 ceph-mon[67070]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:40:58.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:58 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/3394819236' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:40:59.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:58 smithi012 ceph-mon[27236]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:40:59.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:58 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/3394819236' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:40:59.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 166429 12.24 MB/sec execute 213 sec latency 62.141 ms 2024-04-14T13:40:59.889 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:59 smithi008 ceph-mon[67070]: pgmap v8: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.0 MiB/s wr, 386 op/s 2024-04-14T13:40:59.890 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:59 smithi008 ceph-mon[67070]: from='client.34144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:40:59.890 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:59 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/2034685531' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:40:59.890 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:59 smithi008 ceph-mon[67070]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:59.890 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:59 smithi008 ceph-mon[67070]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:59.890 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:59 smithi008 ceph-mon[67070]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:59.890 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:59 smithi008 ceph-mon[67070]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:40:59.891 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:59 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:40:59.891 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:40:59 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:41:00.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:59 smithi012 ceph-mon[27236]: pgmap v8: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.0 MiB/s wr, 386 op/s 2024-04-14T13:41:00.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:59 smithi012 ceph-mon[27236]: from='client.34144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:41:00.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:59 smithi012 ceph-mon[27236]: from='client.? 172.21.15.8:0/2034685531' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:41:00.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:59 smithi012 ceph-mon[27236]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:00.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:59 smithi012 ceph-mon[27236]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:00.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:59 smithi012 ceph-mon[27236]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:00.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:59 smithi012 ceph-mon[27236]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:00.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:59 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:00.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:40:59 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:41:00.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 166651 12.24 MB/sec execute 214 sec latency 57.109 ms 2024-04-14T13:41:00.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:00 smithi008 ceph-mon[67070]: Updating smithi008:/etc/ceph/ceph.conf 2024-04-14T13:41:00.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:00 smithi008 ceph-mon[67070]: Updating smithi012:/etc/ceph/ceph.conf 2024-04-14T13:41:00.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:00 smithi008 ceph-mon[67070]: Updating smithi008:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T13:41:00.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:00 smithi008 ceph-mon[67070]: Updating smithi012:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T13:41:00.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:00 smithi008 ceph-mon[67070]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:41:00.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:00 smithi008 ceph-mon[67070]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:00.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:00 smithi008 ceph-mon[67070]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:00.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:00 smithi008 ceph-mon[67070]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:00.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:00 smithi008 ceph-mon[67070]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:00.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:00 smithi008 ceph-mon[67070]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:00.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:00 smithi008 ceph-mon[67070]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:01.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:00 smithi012 ceph-mon[27236]: Updating smithi008:/etc/ceph/ceph.conf 2024-04-14T13:41:01.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:00 smithi012 ceph-mon[27236]: Updating smithi012:/etc/ceph/ceph.conf 2024-04-14T13:41:01.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:00 smithi012 ceph-mon[27236]: Updating smithi008:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T13:41:01.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:00 smithi012 ceph-mon[27236]: Updating smithi012:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T13:41:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:00 smithi012 ceph-mon[27236]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:41:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:00 smithi012 ceph-mon[27236]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:00 smithi012 ceph-mon[27236]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:00 smithi012 ceph-mon[27236]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:00 smithi012 ceph-mon[27236]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:00 smithi012 ceph-mon[27236]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:00 smithi012 ceph-mon[27236]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:01.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 166966 12.23 MB/sec execute 215 sec latency 75.824 ms 2024-04-14T13:41:01.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:01 smithi008 ceph-mon[67070]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:41:01.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:01 smithi008 ceph-mon[67070]: pgmap v9: 97 pgs: 97 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.9 MiB/s wr, 599 op/s 2024-04-14T13:41:01.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:01 smithi008 ceph-mon[67070]: Updating smithi008:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T13:41:01.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:01 smithi008 ceph-mon[67070]: Updating smithi012:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T13:41:01.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:01 smithi008 ceph-mon[67070]: pgmap v10: 97 pgs: 97 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 6.2 MiB/s wr, 622 op/s 2024-04-14T13:41:01.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:01 smithi008 ceph-mon[67070]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T13:41:01.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:01 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:41:01.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:01 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:41:01.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:01 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T13:41:02.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:01 smithi012 ceph-mon[27236]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:41:02.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:01 smithi012 ceph-mon[27236]: pgmap v9: 97 pgs: 97 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.9 MiB/s wr, 599 op/s 2024-04-14T13:41:02.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:01 smithi012 ceph-mon[27236]: Updating smithi008:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T13:41:02.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:01 smithi012 ceph-mon[27236]: Updating smithi012:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T13:41:02.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:01 smithi012 ceph-mon[27236]: pgmap v10: 97 pgs: 97 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 6.2 MiB/s wr, 622 op/s 2024-04-14T13:41:02.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:01 smithi012 ceph-mon[27236]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T13:41:02.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:01 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:41:02.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:01 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:41:02.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:01 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T13:41:02.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 167348 12.21 MB/sec execute 216 sec latency 56.568 ms 2024-04-14T13:41:03.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 167645 12.19 MB/sec execute 217 sec latency 39.727 ms 2024-04-14T13:41:03.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:03 smithi008 ceph-mon[67070]: pgmap v11: 97 pgs: 97 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 6.2 MiB/s wr, 622 op/s 2024-04-14T13:41:03.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:03 smithi008 ceph-mon[67070]: Upgrade: Updating mon.smithi012 2024-04-14T13:41:03.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:03 smithi008 ceph-mon[67070]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:03.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:41:03.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:41:03.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:03.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:03 smithi008 ceph-mon[67070]: Deploying daemon mon.smithi012 on smithi012 2024-04-14T13:41:03.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:41:03.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:03 smithi012 ceph-mon[27236]: pgmap v11: 97 pgs: 97 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 6.2 MiB/s wr, 622 op/s 2024-04-14T13:41:03.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:03 smithi012 ceph-mon[27236]: Upgrade: Updating mon.smithi012 2024-04-14T13:41:03.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:03 smithi012 ceph-mon[27236]: from='mgr.34100 ' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:03.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:03 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:41:03.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:03 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:41:03.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:03 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:03.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:03 smithi012 ceph-mon[27236]: Deploying daemon mon.smithi012 on smithi012 2024-04-14T13:41:03.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:03 smithi012 ceph-mon[27236]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:41:04.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 167906 12.18 MB/sec execute 218 sec latency 44.049 ms 2024-04-14T13:41:04.954 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:04 smithi012 systemd[1]: Stopping Ceph mon.smithi012 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:41:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:04 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi012[27217]: 2024-04-14T13:41:04.952+0000 7f3b5623e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi012 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:41:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:04 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi012[27217]: 2024-04-14T13:41:04.952+0000 7f3b5623e700 -1 mon.smithi012@1(peon) e2 *** Got Signal Terminated *** 2024-04-14T13:41:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:05 smithi012 podman[53136]: 2024-04-14 13:41:05.05693636 +0000 UTC m=+0.251272925 container died f6cd483dfa3178cbe6380bf4ad87afa6e9e11eb6ae3a6142ee242144f6dfa9b5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi012, name=centos-stream, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, version=8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, maintainer=Guillaume Abrioux , vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, architecture=x86_64, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD) 2024-04-14T13:41:05.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 168147 12.14 MB/sec execute 219 sec latency 107.186 ms 2024-04-14T13:41:06.135 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:05 smithi012 podman[53136]: 2024-04-14 13:41:05.828114838 +0000 UTC m=+1.022451393 container cleanup f6cd483dfa3178cbe6380bf4ad87afa6e9e11eb6ae3a6142ee242144f6dfa9b5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi012, version=8, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, distribution-scope=public, com.redhat.component=centos-stream-container, io.openshift.expose-services=, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git) 2024-04-14T13:41:06.135 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:05 smithi012 bash[53136]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi012 2024-04-14T13:41:06.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 168489 12.11 MB/sec execute 220 sec latency 68.464 ms 2024-04-14T13:41:06.542 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:06 smithi012 podman[53150]: 2024-04-14 13:41:06.132249915 +0000 UTC m=+1.072541406 container remove f6cd483dfa3178cbe6380bf4ad87afa6e9e11eb6ae3a6142ee242144f6dfa9b5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi012, GIT_BRANCH=HEAD, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, version=8, io.openshift.expose-services=, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True) 2024-04-14T13:41:06.542 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:06 smithi012 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@mon.smithi012.service: Deactivated successfully. 2024-04-14T13:41:06.542 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:06 smithi012 systemd[1]: Stopped Ceph mon.smithi012 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:41:06.542 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:06 smithi012 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@mon.smithi012.service: Consumed 7.422s CPU time. 2024-04-14T13:41:06.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:06 smithi012 systemd[1]: Starting Ceph mon.smithi012 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:41:06.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:06 smithi012 podman[53271]: 2024-04-14 13:41:06.677816418 +0000 UTC m=+0.023370707 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:41:07.087 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:06 smithi012 podman[53271]: 2024-04-14 13:41:06.808937357 +0000 UTC m=+0.154491627 container create d498dec29c4f3f1cade97459209d271cb28a9089501bbaca04826f66f4e2cf42 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi012, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-14T13:41:07.087 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 podman[53271]: 2024-04-14 13:41:07.053550719 +0000 UTC m=+0.399104987 container init d498dec29c4f3f1cade97459209d271cb28a9089501bbaca04826f66f4e2cf42 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi012, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-a971fd5) 2024-04-14T13:41:07.087 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 podman[53271]: 2024-04-14 13:41:07.057292145 +0000 UTC m=+0.402846422 container start d498dec29c4f3f1cade97459209d271cb28a9089501bbaca04826f66f4e2cf42 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi012, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_CLEAN=True, RELEASE=main-a971fd5, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T13:41:07.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 168720 12.08 MB/sec execute 221 sec latency 73.618 ms 2024-04-14T13:41:07.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: set uid:gid to 167:167 (ceph:ceph) 2024-04-14T13:41:07.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev), process ceph-mon, pid 2 2024-04-14T13:41:07.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: pidfile_write: ignore empty --pid-file 2024-04-14T13:41:07.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: load: jerasure load: lrc 2024-04-14T13:41:07.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: RocksDB version: 7.9.2 2024-04-14T13:41:07.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Git sha 0 2024-04-14T13:41:07.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Compile date 2024-04-06 20:28:24 2024-04-14T13:41:07.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: DB SUMMARY 2024-04-14T13:41:07.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: DB Session ID: 4YJP2GCCII5E4I3VC1PU 2024-04-14T13:41:07.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: CURRENT file: CURRENT 2024-04-14T13:41:07.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: IDENTITY file: IDENTITY 2024-04-14T13:41:07.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: MANIFEST file: MANIFEST-000009 size: 903 Bytes 2024-04-14T13:41:07.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi012/store.db dir, Total Num: 1, files: 000024.sst 2024-04-14T13:41:07.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi012/store.db: 000022.log size: 7583371 ; 2024-04-14T13:41:07.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.error_if_exists: 0 2024-04-14T13:41:07.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.create_if_missing: 0 2024-04-14T13:41:07.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.paranoid_checks: 1 2024-04-14T13:41:07.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-14T13:41:07.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-14T13:41:07.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-14T13:41:07.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.env: 0x55af4a315c60 2024-04-14T13:41:07.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.fs: PosixFileSystem 2024-04-14T13:41:07.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.info_log: 0x55af4b089aa0 2024-04-14T13:41:07.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_file_opening_threads: 16 2024-04-14T13:41:07.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.statistics: (nil) 2024-04-14T13:41:07.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.use_fsync: 0 2024-04-14T13:41:07.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_log_file_size: 0 2024-04-14T13:41:07.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-14T13:41:07.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-14T13:41:07.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.keep_log_file_num: 1000 2024-04-14T13:41:07.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.recycle_log_file_num: 0 2024-04-14T13:41:07.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.allow_fallocate: 1 2024-04-14T13:41:07.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.allow_mmap_reads: 0 2024-04-14T13:41:07.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.allow_mmap_writes: 0 2024-04-14T13:41:07.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.use_direct_reads: 0 2024-04-14T13:41:07.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-14T13:41:07.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.create_missing_column_families: 0 2024-04-14T13:41:07.554 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.db_log_dir: 2024-04-14T13:41:07.554 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.wal_dir: 2024-04-14T13:41:07.554 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-14T13:41:07.555 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-14T13:41:07.555 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-14T13:41:07.555 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-14T13:41:07.555 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-14T13:41:07.555 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-14T13:41:07.556 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.advise_random_on_open: 1 2024-04-14T13:41:07.556 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.db_write_buffer_size: 0 2024-04-14T13:41:07.556 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.write_buffer_manager: 0x55af4b05a1e0 2024-04-14T13:41:07.556 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-14T13:41:07.556 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-14T13:41:07.556 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-14T13:41:07.556 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.rate_limiter: (nil) 2024-04-14T13:41:07.556 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-14T13:41:07.556 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.wal_recovery_mode: 2 2024-04-14T13:41:07.556 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.enable_thread_tracking: 0 2024-04-14T13:41:07.556 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.enable_pipelined_write: 0 2024-04-14T13:41:07.556 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.unordered_write: 0 2024-04-14T13:41:07.556 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-14T13:41:07.556 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-14T13:41:07.556 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-14T13:41:07.556 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-14T13:41:07.557 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.row_cache: None 2024-04-14T13:41:07.557 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.wal_filter: None 2024-04-14T13:41:07.557 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-14T13:41:07.557 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.allow_ingest_behind: 0 2024-04-14T13:41:07.557 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.two_write_queues: 0 2024-04-14T13:41:07.557 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.manual_wal_flush: 0 2024-04-14T13:41:07.557 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.wal_compression: 0 2024-04-14T13:41:07.557 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.atomic_flush: 0 2024-04-14T13:41:07.557 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-14T13:41:07.557 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-14T13:41:07.557 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-14T13:41:07.557 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.log_readahead_size: 0 2024-04-14T13:41:07.557 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-14T13:41:07.557 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.best_efforts_recovery: 0 2024-04-14T13:41:07.557 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-14T13:41:07.558 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-14T13:41:07.558 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.allow_data_in_errors: 0 2024-04-14T13:41:07.558 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.db_host_id: __hostname__ 2024-04-14T13:41:07.558 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-14T13:41:07.558 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_background_jobs: 2 2024-04-14T13:41:07.558 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_background_compactions: -1 2024-04-14T13:41:07.558 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_subcompactions: 1 2024-04-14T13:41:07.558 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-14T13:41:07.558 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-14T13:41:07.558 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-14T13:41:07.558 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_total_wal_size: 0 2024-04-14T13:41:07.559 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-14T13:41:07.559 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-14T13:41:07.559 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-14T13:41:07.559 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-14T13:41:07.559 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_open_files: -1 2024-04-14T13:41:07.559 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.bytes_per_sync: 0 2024-04-14T13:41:07.559 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-14T13:41:07.559 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-14T13:41:07.559 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compaction_readahead_size: 0 2024-04-14T13:41:07.559 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_background_flushes: -1 2024-04-14T13:41:07.559 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Compression algorithms supported: 2024-04-14T13:41:07.559 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-14T13:41:07.559 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: kZSTD supported: 0 2024-04-14T13:41:07.559 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: kXpressCompression supported: 0 2024-04-14T13:41:07.559 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: kLZ4HCCompression supported: 1 2024-04-14T13:41:07.560 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: kZlibCompression supported: 1 2024-04-14T13:41:07.560 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: kSnappyCompression supported: 1 2024-04-14T13:41:07.560 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: kLZ4Compression supported: 1 2024-04-14T13:41:07.560 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: kBZip2Compression supported: 0 2024-04-14T13:41:07.560 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-14T13:41:07.560 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-14T13:41:07.560 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi012/store.db/MANIFEST-000009 2024-04-14T13:41:07.560 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-14T13:41:07.560 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-14T13:41:07.560 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.merge_operator: 2024-04-14T13:41:07.560 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compaction_filter: None 2024-04-14T13:41:07.560 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compaction_filter_factory: None 2024-04-14T13:41:07.560 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.sst_partitioner_factory: None 2024-04-14T13:41:07.560 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-14T13:41:07.560 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-14T13:41:07.560 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55af4b089c80) 2024-04-14T13:41:07.561 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: cache_index_and_filter_blocks: 1 2024-04-14T13:41:07.561 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-14T13:41:07.561 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-14T13:41:07.561 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: pin_top_level_index_and_filter: 1 2024-04-14T13:41:07.561 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: index_type: 0 2024-04-14T13:41:07.561 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: data_block_index_type: 0 2024-04-14T13:41:07.561 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: index_shortening: 1 2024-04-14T13:41:07.561 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-14T13:41:07.561 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: checksum: 4 2024-04-14T13:41:07.561 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: no_block_cache: 0 2024-04-14T13:41:07.561 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: block_cache: 0x55af4b08d090 2024-04-14T13:41:07.561 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: block_cache_name: BinnedLRUCache 2024-04-14T13:41:07.561 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: block_cache_options: 2024-04-14T13:41:07.561 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: capacity : 536870912 2024-04-14T13:41:07.562 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: num_shard_bits : 4 2024-04-14T13:41:07.562 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: strict_capacity_limit : 0 2024-04-14T13:41:07.562 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: high_pri_pool_ratio: 0.000 2024-04-14T13:41:07.562 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: block_cache_compressed: (nil) 2024-04-14T13:41:07.562 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: persistent_cache: (nil) 2024-04-14T13:41:07.562 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: block_size: 4096 2024-04-14T13:41:07.562 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: block_size_deviation: 10 2024-04-14T13:41:07.562 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: block_restart_interval: 16 2024-04-14T13:41:07.562 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: index_block_restart_interval: 1 2024-04-14T13:41:07.562 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: metadata_block_size: 4096 2024-04-14T13:41:07.562 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: partition_filters: 0 2024-04-14T13:41:07.562 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: use_delta_encoding: 1 2024-04-14T13:41:07.562 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: filter_policy: bloomfilter 2024-04-14T13:41:07.562 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: whole_key_filtering: 1 2024-04-14T13:41:07.562 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: verify_compression: 0 2024-04-14T13:41:07.563 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: read_amp_bytes_per_bit: 0 2024-04-14T13:41:07.563 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: format_version: 5 2024-04-14T13:41:07.563 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: enable_index_compression: 1 2024-04-14T13:41:07.563 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: block_align: 0 2024-04-14T13:41:07.563 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: max_auto_readahead_size: 262144 2024-04-14T13:41:07.563 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: prepopulate_block_cache: 0 2024-04-14T13:41:07.563 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: initial_auto_readahead_size: 8192 2024-04-14T13:41:07.563 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: num_file_reads_for_auto_readahead: 2 2024-04-14T13:41:07.563 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.write_buffer_size: 33554432 2024-04-14T13:41:07.563 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_write_buffer_number: 2 2024-04-14T13:41:07.563 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compression: NoCompression 2024-04-14T13:41:07.563 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.bottommost_compression: Disabled 2024-04-14T13:41:07.563 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.prefix_extractor: nullptr 2024-04-14T13:41:07.563 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-14T13:41:07.563 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.num_levels: 7 2024-04-14T13:41:07.563 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-14T13:41:07.564 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-14T13:41:07.564 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-14T13:41:07.564 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-14T13:41:07.564 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-14T13:41:07.564 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-14T13:41:07.564 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-14T13:41:07.564 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-14T13:41:07.564 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-14T13:41:07.564 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-14T13:41:07.564 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-14T13:41:07.564 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-14T13:41:07.564 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-14T13:41:07.564 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compression_opts.level: 32767 2024-04-14T13:41:07.564 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compression_opts.strategy: 0 2024-04-14T13:41:07.564 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-14T13:41:07.565 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-14T13:41:07.565 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-14T13:41:07.565 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-14T13:41:07.565 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compression_opts.enabled: false 2024-04-14T13:41:07.565 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-14T13:41:07.565 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-14T13:41:07.565 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-14T13:41:07.565 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-14T13:41:07.565 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.target_file_size_base: 67108864 2024-04-14T13:41:07.565 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-14T13:41:07.565 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-14T13:41:07.565 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-14T13:41:07.565 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-14T13:41:07.565 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-14T13:41:07.565 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-14T13:41:07.565 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-14T13:41:07.565 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-14T13:41:07.566 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-14T13:41:07.566 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-14T13:41:07.566 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-14T13:41:07.566 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-14T13:41:07.566 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-14T13:41:07.566 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-14T13:41:07.566 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.arena_block_size: 1048576 2024-04-14T13:41:07.566 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-14T13:41:07.566 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-14T13:41:07.566 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.disable_auto_compactions: 0 2024-04-14T13:41:07.566 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-14T13:41:07.566 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-14T13:41:07.566 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-14T13:41:07.566 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-14T13:41:07.567 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-14T13:41:07.567 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-14T13:41:07.567 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-14T13:41:07.567 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-14T13:41:07.567 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-14T13:41:07.567 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-14T13:41:07.567 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-14T13:41:07.567 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.inplace_update_support: 0 2024-04-14T13:41:07.567 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-14T13:41:07.567 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-14T13:41:07.567 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-14T13:41:07.567 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-14T13:41:07.567 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.bloom_locality: 0 2024-04-14T13:41:07.567 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.max_successive_merges: 0 2024-04-14T13:41:07.567 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-14T13:41:07.567 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.paranoid_file_checks: 0 2024-04-14T13:41:07.567 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.force_consistency_checks: 1 2024-04-14T13:41:07.568 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.report_bg_io_stats: 0 2024-04-14T13:41:07.568 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.ttl: 2592000 2024-04-14T13:41:07.568 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-14T13:41:07.568 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-14T13:41:07.568 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-14T13:41:07.568 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.enable_blob_files: false 2024-04-14T13:41:07.568 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.min_blob_size: 0 2024-04-14T13:41:07.568 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.blob_file_size: 268435456 2024-04-14T13:41:07.568 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-14T13:41:07.568 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-14T13:41:07.568 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-14T13:41:07.568 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-14T13:41:07.568 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-14T13:41:07.568 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.blob_file_starting_level: 0 2024-04-14T13:41:07.568 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-14T13:41:07.568 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 24.sst 2024-04-14T13:41:07.568 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-14T13:41:07.569 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi012/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 26, last_sequence is 12197, log_number is 22,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-14T13:41:07.569 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 22 2024-04-14T13:41:07.569 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1d52b2c7-9f0f-4c49-9e9d-dbf7b1076856 2024-04-14T13:41:07.569 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713102067102485, "job": 1, "event": "recovery_started", "wal_files": [22]} 2024-04-14T13:41:07.569 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #22 mode 2 2024-04-14T13:41:07.569 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713102067122047, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 27, "file_size": 4627374, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12202, "largest_seqno": 14114, "table_properties": {"data_size": 4619121, "index_size": 5287, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1925, "raw_key_size": 17216, "raw_average_key_size": 23, "raw_value_size": 4602964, "raw_average_value_size": 6153, "num_data_blocks": 240, "num_entries": 748, "num_filter_entries": 748, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713102067, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1d52b2c7-9f0f-4c49-9e9d-dbf7b1076856", "db_session_id": "4YJP2GCCII5E4I3VC1PU", "orig_file_number": 27, "seqno_to_time_mapping": "N/A"}} 2024-04-14T13:41:07.569 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713102067122177, "job": 1, "event": "recovery_finished"} 2024-04-14T13:41:07.569 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: [db/version_set.cc:5047] Creating manifest 29 2024-04-14T13:41:07.569 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-14T13:41:07.569 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi012/store.db/000022.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-14T13:41:07.569 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55af4b17c000 2024-04-14T13:41:07.569 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: DB pointer 0x55af4b166000 2024-04-14T13:41:07.569 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-14T13:41:07.569 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-14T13:41:07.569 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: ** DB Stats ** 2024-04-14T13:41:07.569 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T13:41:07.569 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-14T13:41:07.570 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T13:41:07.570 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T13:41:07.570 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-14T13:41:07.570 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T13:41:07.570 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T13:41:07.570 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: 2024-04-14T13:41:07.570 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: ** Compaction Stats [default] ** 2024-04-14T13:41:07.570 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T13:41:07.570 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-14T13:41:07.570 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: L0 1/0 4.41 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 487.1 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-04-14T13:41:07.570 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: L6 1/0 6.52 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-14T13:41:07.570 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Sum 2/0 10.93 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 487.1 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-04-14T13:41:07.570 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 487.1 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-04-14T13:41:07.570 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: 2024-04-14T13:41:07.570 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: ** Compaction Stats [default] ** 2024-04-14T13:41:07.571 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T13:41:07.571 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T13:41:07.571 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 487.1 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-04-14T13:41:07.571 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: 2024-04-14T13:41:07.571 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-14T13:41:07.571 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: 2024-04-14T13:41:07.571 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T13:41:07.571 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-04-14T13:41:07.571 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T13:41:07.571 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T13:41:07.571 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T13:41:07.571 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T13:41:07.571 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Cumulative compaction: 0.00 GB write, 133.03 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T13:41:07.571 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Interval compaction: 0.00 GB write, 133.03 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T13:41:07.571 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T13:41:07.571 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Block cache BinnedLRUCache@0x55af4b08d090#2 capacity: 512.00 MB usage: 8.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-14T13:41:07.572 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,6.08 KB,0.00115931%) Misc(1,0.00 KB,0%) 2024-04-14T13:41:07.572 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: 2024-04-14T13:41:07.572 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T13:41:07.572 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: starting mon.smithi012 rank 1 at public addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] at bind addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi012 fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:41:07.572 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: mon.smithi012@-1(???) e2 preinit fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:41:07.572 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: mon.smithi012@-1(???).mds e7 new map 2024-04-14T13:41:07.572 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: mon.smithi012@-1(???).mds e7 print_map 2024-04-14T13:41:07.572 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: e7 2024-04-14T13:41:07.572 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-14T13:41:07.572 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-14T13:41:07.572 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: legacy client fscid: 1 2024-04-14T13:41:07.572 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: 2024-04-14T13:41:07.572 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Filesystem 'foofs' (1) 2024-04-14T13:41:07.572 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: fs_name foofs 2024-04-14T13:41:07.572 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: epoch 5 2024-04-14T13:41:07.572 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-14T13:41:07.573 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: created 2024-04-14T13:34:52.552946+0000 2024-04-14T13:41:07.573 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: modified 2024-04-14T13:34:58.569998+0000 2024-04-14T13:41:07.573 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: tableserver 0 2024-04-14T13:41:07.573 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: root 0 2024-04-14T13:41:07.573 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: session_timeout 60 2024-04-14T13:41:07.573 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: session_autoclose 300 2024-04-14T13:41:07.573 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: max_file_size 1099511627776 2024-04-14T13:41:07.573 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: max_xattr_size 65536 2024-04-14T13:41:07.573 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: required_client_features {} 2024-04-14T13:41:07.573 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: last_failure 0 2024-04-14T13:41:07.573 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: last_failure_osd_epoch 0 2024-04-14T13:41:07.573 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-14T13:41:07.573 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: max_mds 1 2024-04-14T13:41:07.573 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: in 0 2024-04-14T13:41:07.573 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: up {0=14506} 2024-04-14T13:41:07.574 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: failed 2024-04-14T13:41:07.574 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: damaged 2024-04-14T13:41:07.574 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: stopped 2024-04-14T13:41:07.574 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: data_pools [3] 2024-04-14T13:41:07.574 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: metadata_pool 2 2024-04-14T13:41:07.574 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: inline_data disabled 2024-04-14T13:41:07.574 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: balancer 2024-04-14T13:41:07.574 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: bal_rank_mask -1 2024-04-14T13:41:07.574 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: standby_count_wanted 1 2024-04-14T13:41:07.574 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: qdb_cluster leader: 0 members: 2024-04-14T13:41:07.574 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: [mds.foofs.smithi008.wqbvfi{0:14506} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.8:6834/785972446,v1:172.21.15.8:6835/785972446] compat {c=[1],r=[1],i=[7ff]}] 2024-04-14T13:41:07.574 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: 2024-04-14T13:41:07.574 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: 2024-04-14T13:41:07.574 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: Standby daemons: 2024-04-14T13:41:07.574 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: 2024-04-14T13:41:07.575 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout: [mds.foofs.smithi012.vjovzp{-1:24309} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.12:6832/1210020061,v1:172.21.15.12:6833/1210020061] compat {c=[1],r=[1],i=[7ff]}] 2024-04-14T13:41:07.575 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: mon.smithi012@-1(???).osd e73 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-14T13:41:07.575 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: mon.smithi012@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T13:41:07.575 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: mon.smithi012@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T13:41:07.575 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: mon.smithi012@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T13:41:07.575 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 ceph-mon[53289]: mon.smithi012@-1(???).paxosservice(auth 1..30) refresh upgraded, format 0 -> 3 2024-04-14T13:41:07.575 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 bash[53271]: d498dec29c4f3f1cade97459209d271cb28a9089501bbaca04826f66f4e2cf42 2024-04-14T13:41:07.575 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:07 smithi012 systemd[1]: Started Ceph mon.smithi012 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:41:08.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 168997 12.07 MB/sec execute 222 sec latency 47.542 ms 2024-04-14T13:41:08.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:08 smithi012 ceph-mon[53289]: pgmap v12: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 31 KiB/s rd, 8.5 MiB/s wr, 835 op/s 2024-04-14T13:41:08.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:08 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:41:08.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:08 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:41:08.822 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:08 smithi008 ceph-mon[67070]: pgmap v12: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 31 KiB/s rd, 8.5 MiB/s wr, 835 op/s 2024-04-14T13:41:08.822 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:08 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:41:08.822 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:08 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:41:09.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 169303 12.05 MB/sec execute 223 sec latency 55.570 ms 2024-04-14T13:41:09.834 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: pgmap v14: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.1 MiB/s wr, 506 op/s 2024-04-14T13:41:09.835 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:41:09.835 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:41:09.835 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: mon.smithi008 calling monitor election 2024-04-14T13:41:09.835 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: mon.smithi012 calling monitor election 2024-04-14T13:41:09.835 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: mon.smithi008 is new leader, mons smithi008,smithi012 in quorum (ranks 0,1) 2024-04-14T13:41:09.835 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: monmap epoch 3 2024-04-14T13:41:09.835 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:41:09.835 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: last_changed 2024-04-14T13:41:08.646185+0000 2024-04-14T13:41:09.835 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: created 2024-04-14T13:27:27.599436+0000 2024-04-14T13:41:09.835 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: min_mon_release 19 (squid) 2024-04-14T13:41:09.835 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: election_strategy: 1 2024-04-14T13:41:09.835 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: 0: [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon.smithi008 2024-04-14T13:41:09.835 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: 1: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.smithi012 2024-04-14T13:41:09.835 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: fsmap foofs:1 {0=foofs.smithi008.wqbvfi=up:active} 1 up:standby 2024-04-14T13:41:09.836 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: osdmap e73: 8 total, 8 up, 8 in 2024-04-14T13:41:09.836 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: mgrmap e41: smithi008.yemlhr(active, since 20s), standbys: smithi012.ksvnxm 2024-04-14T13:41:09.836 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T13:41:09.836 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:41:09.836 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: daemon grafana.smithi008 on smithi008 is in error state 2024-04-14T13:41:09.836 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:09.836 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:09.836 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:09 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:41:09.877 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: pgmap v14: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.1 MiB/s wr, 506 op/s 2024-04-14T13:41:09.877 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-14T13:41:09.877 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mon metadata", "id": "smithi012"}]: dispatch 2024-04-14T13:41:09.877 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: mon.smithi008 calling monitor election 2024-04-14T13:41:09.877 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: mon.smithi012 calling monitor election 2024-04-14T13:41:09.877 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: mon.smithi008 is new leader, mons smithi008,smithi012 in quorum (ranks 0,1) 2024-04-14T13:41:09.877 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: monmap epoch 3 2024-04-14T13:41:09.877 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 2024-04-14T13:41:09.878 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: last_changed 2024-04-14T13:41:08.646185+0000 2024-04-14T13:41:09.878 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: created 2024-04-14T13:27:27.599436+0000 2024-04-14T13:41:09.878 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: min_mon_release 19 (squid) 2024-04-14T13:41:09.878 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: election_strategy: 1 2024-04-14T13:41:09.878 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: 0: [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon.smithi008 2024-04-14T13:41:09.878 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: 1: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.smithi012 2024-04-14T13:41:09.878 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: fsmap foofs:1 {0=foofs.smithi008.wqbvfi=up:active} 1 up:standby 2024-04-14T13:41:09.878 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: osdmap e73: 8 total, 8 up, 8 in 2024-04-14T13:41:09.878 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: mgrmap e41: smithi008.yemlhr(active, since 20s), standbys: smithi012.ksvnxm 2024-04-14T13:41:09.878 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T13:41:09.878 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:41:09.878 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: daemon grafana.smithi008 on smithi008 is in error state 2024-04-14T13:41:09.878 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:09.878 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:09.878 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:09 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:41:10.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 169723 12.04 MB/sec execute 224 sec latency 52.424 ms 2024-04-14T13:41:11.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 170032 12.02 MB/sec execute 225 sec latency 45.261 ms 2024-04-14T13:41:11.667 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:11 smithi008 ceph-mon[67070]: pgmap v15: 97 pgs: 97 active+clean; 326 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.2 MiB/s wr, 416 op/s 2024-04-14T13:41:12.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:11 smithi012 ceph-mon[53289]: pgmap v15: 97 pgs: 97 active+clean; 326 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.2 MiB/s wr, 416 op/s 2024-04-14T13:41:12.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 170291 12.01 MB/sec execute 226 sec latency 60.493 ms 2024-04-14T13:41:13.195 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:13 smithi008 ceph-mon[67070]: pgmap v16: 97 pgs: 97 active+clean; 326 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.7 MiB/s wr, 367 op/s 2024-04-14T13:41:13.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 170545 12.01 MB/sec execute 227 sec latency 64.964 ms 2024-04-14T13:41:13.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:13 smithi012 ceph-mon[53289]: pgmap v16: 97 pgs: 97 active+clean; 326 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.7 MiB/s wr, 367 op/s 2024-04-14T13:41:14.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:14 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:14.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:14 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:14.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 170801 11.97 MB/sec execute 228 sec latency 69.469 ms 2024-04-14T13:41:14.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:14 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:14.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:14 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:15.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 171061 11.96 MB/sec execute 229 sec latency 59.313 ms 2024-04-14T13:41:15.545 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:15 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:15.545 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:15 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:15.545 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:15 smithi008 ceph-mon[67070]: pgmap v17: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.6 MiB/s wr, 541 op/s 2024-04-14T13:41:15.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:15 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:15.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:15 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:15.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:15 smithi012 ceph-mon[53289]: pgmap v17: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.6 MiB/s wr, 541 op/s 2024-04-14T13:41:16.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 171278 11.93 MB/sec execute 230 sec latency 62.761 ms 2024-04-14T13:41:17.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 171634 11.92 MB/sec execute 231 sec latency 37.210 ms 2024-04-14T13:41:17.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:17 smithi012 ceph-mon[53289]: pgmap v18: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.5 MiB/s wr, 353 op/s 2024-04-14T13:41:17.824 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:17 smithi008 ceph-mon[67070]: pgmap v18: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.5 MiB/s wr, 353 op/s 2024-04-14T13:41:18.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 172029 11.90 MB/sec execute 232 sec latency 39.475 ms 2024-04-14T13:41:18.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:18 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:41:18.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:18 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:18.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:18 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:18.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:18 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:18.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:18 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:41:18.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:18 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:18.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:18 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:18.838 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:18 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:41:18.838 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:18 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:18.838 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:18 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:18.838 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:18 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:18.838 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:18 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:41:18.839 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:18 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:18.839 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:18 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:19.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 172343 11.88 MB/sec execute 233 sec latency 66.386 ms 2024-04-14T13:41:19.856 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:19 smithi008 ceph-mon[67070]: pgmap v19: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.5 MiB/s wr, 355 op/s 2024-04-14T13:41:19.856 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:19 smithi008 ceph-mon[67070]: Reconfiguring mon.smithi008 (monmap changed)... 2024-04-14T13:41:19.856 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:19 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:41:19.856 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:19 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:41:19.856 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:19 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:19.856 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:19 smithi008 ceph-mon[67070]: Reconfiguring daemon mon.smithi008 on smithi008 2024-04-14T13:41:20.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:19 smithi012 ceph-mon[53289]: pgmap v19: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.5 MiB/s wr, 355 op/s 2024-04-14T13:41:20.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:19 smithi012 ceph-mon[53289]: Reconfiguring mon.smithi008 (monmap changed)... 2024-04-14T13:41:20.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:19 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:41:20.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:19 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:41:20.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:19 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:20.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:19 smithi012 ceph-mon[53289]: Reconfiguring daemon mon.smithi008 on smithi008 2024-04-14T13:41:20.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 172653 11.88 MB/sec execute 234 sec latency 47.825 ms 2024-04-14T13:41:21.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 172993 11.85 MB/sec execute 235 sec latency 47.931 ms 2024-04-14T13:41:21.650 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:21 smithi008 ceph-mon[67070]: pgmap v20: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.4 MiB/s wr, 550 op/s 2024-04-14T13:41:21.650 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:21 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:21.651 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:21 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:21.651 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:21 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.yemlhr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:41:21.651 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:21 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:41:21.651 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:21 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:21.993 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:21 smithi012 ceph-mon[53289]: pgmap v20: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.4 MiB/s wr, 550 op/s 2024-04-14T13:41:21.993 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:21 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:21.993 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:21 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:21.993 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:21 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.yemlhr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:41:21.993 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:21 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:41:21.994 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:21 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:22.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 173333 11.83 MB/sec execute 236 sec latency 41.857 ms 2024-04-14T13:41:22.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:22 smithi008 ceph-mon[67070]: Reconfiguring mgr.smithi008.yemlhr (monmap changed)... 2024-04-14T13:41:22.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:22 smithi008 ceph-mon[67070]: Reconfiguring daemon mgr.smithi008.yemlhr on smithi008 2024-04-14T13:41:23.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:22 smithi012 ceph-mon[53289]: Reconfiguring mgr.smithi008.yemlhr (monmap changed)... 2024-04-14T13:41:23.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:22 smithi012 ceph-mon[53289]: Reconfiguring daemon mgr.smithi008.yemlhr on smithi008 2024-04-14T13:41:23.342 INFO:teuthology.orchestra.run.smithi008.stdout: 5 173623 11.83 MB/sec execute 237 sec latency 42.731 ms 2024-04-14T13:41:23.677 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:23 smithi008 ceph-mon[67070]: pgmap v21: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.8 MiB/s wr, 369 op/s 2024-04-14T13:41:24.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:23 smithi012 ceph-mon[53289]: pgmap v21: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.8 MiB/s wr, 369 op/s 2024-04-14T13:41:24.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 173981 11.84 MB/sec execute 238 sec latency 54.004 ms 2024-04-14T13:41:24.952 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:24 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:24.953 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:24 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:24.953 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:24 smithi008 ceph-mon[67070]: Reconfiguring crash.smithi008 (monmap changed)... 2024-04-14T13:41:24.953 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:24 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-14T13:41:24.953 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:24 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:24.953 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:24 smithi008 ceph-mon[67070]: Reconfiguring daemon crash.smithi008 on smithi008 2024-04-14T13:41:25.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:24 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:25.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:24 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:25.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:24 smithi012 ceph-mon[53289]: Reconfiguring crash.smithi008 (monmap changed)... 2024-04-14T13:41:25.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:24 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-14T13:41:25.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:24 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:25.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:24 smithi012 ceph-mon[53289]: Reconfiguring daemon crash.smithi008 on smithi008 2024-04-14T13:41:25.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 174311 11.82 MB/sec execute 239 sec latency 45.948 ms 2024-04-14T13:41:25.960 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:25 smithi008 ceph-mon[67070]: pgmap v22: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.8 MiB/s wr, 570 op/s 2024-04-14T13:41:26.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:25 smithi012 ceph-mon[53289]: pgmap v22: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.8 MiB/s wr, 570 op/s 2024-04-14T13:41:26.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 174634 11.81 MB/sec execute 240 sec latency 36.325 ms 2024-04-14T13:41:27.342 INFO:teuthology.orchestra.run.smithi008.stdout: 5 174964 11.80 MB/sec execute 241 sec latency 34.654 ms 2024-04-14T13:41:27.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:27 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:27.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:27 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:27.481 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:27 smithi008 ceph-mon[67070]: Reconfiguring osd.0 (monmap changed)... 2024-04-14T13:41:27.481 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:27 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T13:41:27.481 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:27 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:27.481 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:27 smithi008 ceph-mon[67070]: Reconfiguring daemon osd.0 on smithi008 2024-04-14T13:41:27.481 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:27 smithi008 ceph-mon[67070]: pgmap v23: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.9 MiB/s wr, 395 op/s 2024-04-14T13:41:27.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:27 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:27.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:27 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:27.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:27 smithi012 ceph-mon[53289]: Reconfiguring osd.0 (monmap changed)... 2024-04-14T13:41:27.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:27 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T13:41:27.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:27 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:27.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:27 smithi012 ceph-mon[53289]: Reconfiguring daemon osd.0 on smithi008 2024-04-14T13:41:27.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:27 smithi012 ceph-mon[53289]: pgmap v23: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.9 MiB/s wr, 395 op/s 2024-04-14T13:41:28.341 INFO:teuthology.orchestra.run.smithi008.stdout: 5 175483 11.80 MB/sec execute 242 sec latency 35.740 ms 2024-04-14T13:41:29.342 INFO:teuthology.orchestra.run.smithi008.stdout: 5 175908 11.79 MB/sec execute 243 sec latency 50.562 ms 2024-04-14T13:41:29.369 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:41:29.757 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:29 smithi008 ceph-mon[67070]: pgmap v24: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.9 MiB/s wr, 395 op/s 2024-04-14T13:41:29.757 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:29.757 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:29.757 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T13:41:29.757 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:30.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:29 smithi012 ceph-mon[53289]: pgmap v24: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.9 MiB/s wr, 395 op/s 2024-04-14T13:41:30.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:30.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:30.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T13:41:30.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:30.342 INFO:teuthology.orchestra.run.smithi008.stdout: 5 176318 11.79 MB/sec execute 244 sec latency 50.464 ms 2024-04-14T13:41:30.453 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:41:30.453 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (3m) 33s ago 12m 16.8M - 0.23.0 ba2b418f427c cfddb9a7f858 2024-04-14T13:41:30.453 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (12m) 33s ago 12m 6405k - 17.2.0 e1d6a67b021e 68b810a9ff93 2024-04-14T13:41:30.453 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (11m) 17s ago 11m 6400k - 17.2.0 e1d6a67b021e 6bc31c51b938 2024-04-14T13:41:30.453 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 33s ago 12m - - 2024-04-14T13:41:30.453 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (6m) 33s ago 6m 38.0M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:41:30.453 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (6m) 17s ago 6m 14.2M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:41:30.453 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (2m) 33s ago 13m 477M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:41:30.453 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (74s) 17s ago 11m 449M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:41:30.453 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (53s) 33s ago 14m 30.4M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 673dd6f4c8d8 2024-04-14T13:41:30.453 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (23s) 17s ago 11m 24.4M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 d498dec29c4f 2024-04-14T13:41:30.453 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (6m) 33s ago 6m 275M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:41:30.454 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (6m) 17s ago 6m 88.4M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:41:30.454 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (2m) 33s ago 12m 14.9M - 1.3.1 1dbe0e931976 646dfca70be4 2024-04-14T13:41:30.454 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (2m) 17s ago 11m 18.4M - 1.3.1 1dbe0e931976 0a60dcd4044c 2024-04-14T13:41:30.454 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (10m) 33s ago 10m 323M 1937M 17.2.0 e1d6a67b021e 3b8cdea7890c 2024-04-14T13:41:30.454 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (10m) 33s ago 10m 488M 1937M 17.2.0 e1d6a67b021e be86642c7790 2024-04-14T13:41:30.454 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (10m) 33s ago 10m 344M 1937M 17.2.0 e1d6a67b021e b1017f00578f 2024-04-14T13:41:30.454 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (9m) 33s ago 9m 510M 1937M 17.2.0 e1d6a67b021e 66fca4b191be 2024-04-14T13:41:30.454 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (9m) 17s ago 9m 570M 2705M 17.2.0 e1d6a67b021e 43b02cb64951 2024-04-14T13:41:30.454 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (8m) 17s ago 8m 362M 2705M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:41:30.454 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (8m) 17s ago 8m 555M 2705M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:41:30.454 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (7m) 17s ago 7m 463M 2705M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:41:30.454 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (82s) 33s ago 11m 64.3M - 2.33.4 514e6a882f6e 12f97f75d620 2024-04-14T13:41:30.763 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:30 smithi008 ceph-mon[67070]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:41:30.763 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:30 smithi008 ceph-mon[67070]: Reconfiguring osd.1 (monmap changed)... 2024-04-14T13:41:30.763 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:30 smithi008 ceph-mon[67070]: Reconfiguring daemon osd.1 on smithi008 2024-04-14T13:41:31.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:30 smithi012 ceph-mon[53289]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:41:31.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:30 smithi012 ceph-mon[53289]: Reconfiguring osd.1 (monmap changed)... 2024-04-14T13:41:31.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:30 smithi012 ceph-mon[53289]: Reconfiguring daemon osd.1 on smithi008 2024-04-14T13:41:31.097 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:41:31.097 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:41:31.098 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:41:31.098 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:41:31.098 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:41:31.098 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:41:31.098 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:41:31.098 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:41:31.098 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T13:41:31.098 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:41:31.098 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:41:31.098 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:41:31.098 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:41:31.098 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:41:31.098 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-14T13:41:31.098 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 4 2024-04-14T13:41:31.099 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:41:31.099 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:41:31.342 INFO:teuthology.orchestra.run.smithi008.stdout: 5 176825 11.78 MB/sec execute 245 sec latency 47.119 ms 2024-04-14T13:41:31.587 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:31 smithi008 ceph-mon[67070]: from='client.44103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:41:31.587 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:31 smithi008 ceph-mon[67070]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:41:31.587 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:31 smithi008 ceph-mon[67070]: pgmap v25: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.2 MiB/s wr, 629 op/s 2024-04-14T13:41:31.587 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:31 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/2878244998' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:41:31.621 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:41:31.621 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:41:31.621 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:41:31.621 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:41:31.621 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-14T13:41:31.621 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-04-14T13:41:31.621 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-04-14T13:41:31.621 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:41:31.622 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "4/23 daemons upgraded", 2024-04-14T13:41:31.622 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading mon daemons", 2024-04-14T13:41:31.622 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:41:31.622 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:41:31.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:31 smithi012 ceph-mon[53289]: from='client.44103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:41:31.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:31 smithi012 ceph-mon[53289]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:41:31.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:31 smithi012 ceph-mon[53289]: pgmap v25: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.2 MiB/s wr, 629 op/s 2024-04-14T13:41:31.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:31 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/2878244998' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:41:32.298 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T13:41:32.298 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:41:32.298 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-14T13:41:32.342 INFO:teuthology.orchestra.run.smithi008.stdout: 5 177166 11.78 MB/sec execute 246 sec latency 30.947 ms 2024-04-14T13:41:32.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:32 smithi008 ceph-mon[67070]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:41:32.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:32 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/278644463' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:41:32.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:32 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:32.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:32 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:32.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:32 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T13:41:32.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:32 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:33.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:32 smithi012 ceph-mon[53289]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:41:33.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:32 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/278644463' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:41:33.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:32 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:33.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:32 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:33.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:32 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T13:41:33.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:32 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:33.342 INFO:teuthology.orchestra.run.smithi008.stdout: 5 177589 11.80 MB/sec execute 247 sec latency 66.515 ms 2024-04-14T13:41:33.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:33 smithi008 ceph-mon[67070]: Reconfiguring osd.2 (monmap changed)... 2024-04-14T13:41:33.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:33 smithi008 ceph-mon[67070]: Reconfiguring daemon osd.2 on smithi008 2024-04-14T13:41:33.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:33 smithi008 ceph-mon[67070]: pgmap v26: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 435 op/s 2024-04-14T13:41:33.803 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:33 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:41:34.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:33 smithi012 ceph-mon[53289]: Reconfiguring osd.2 (monmap changed)... 2024-04-14T13:41:34.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:33 smithi012 ceph-mon[53289]: Reconfiguring daemon osd.2 on smithi008 2024-04-14T13:41:34.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:33 smithi012 ceph-mon[53289]: pgmap v26: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 435 op/s 2024-04-14T13:41:34.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:33 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:41:34.342 INFO:teuthology.orchestra.run.smithi008.stdout: 5 178099 11.81 MB/sec execute 248 sec latency 27.016 ms 2024-04-14T13:41:35.342 INFO:teuthology.orchestra.run.smithi008.stdout: 5 178416 11.79 MB/sec execute 249 sec latency 46.045 ms 2024-04-14T13:41:35.820 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:35 smithi008 ceph-mon[67070]: pgmap v27: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.0 MiB/s wr, 687 op/s 2024-04-14T13:41:35.820 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:35 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:35.820 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:35 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:35.821 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:35 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T13:41:35.821 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:35 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:36.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:35 smithi012 ceph-mon[53289]: pgmap v27: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.0 MiB/s wr, 687 op/s 2024-04-14T13:41:36.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:35 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:36.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:35 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:36.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:35 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T13:41:36.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:35 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:36.342 INFO:teuthology.orchestra.run.smithi008.stdout: 5 178811 11.78 MB/sec execute 250 sec latency 80.346 ms 2024-04-14T13:41:36.829 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:36 smithi008 ceph-mon[67070]: Reconfiguring osd.3 (monmap changed)... 2024-04-14T13:41:36.829 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:36 smithi008 ceph-mon[67070]: Reconfiguring daemon osd.3 on smithi008 2024-04-14T13:41:37.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:36 smithi012 ceph-mon[53289]: Reconfiguring osd.3 (monmap changed)... 2024-04-14T13:41:37.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:36 smithi012 ceph-mon[53289]: Reconfiguring daemon osd.3 on smithi008 2024-04-14T13:41:37.342 INFO:teuthology.orchestra.run.smithi008.stdout: 5 179257 11.77 MB/sec execute 251 sec latency 36.696 ms 2024-04-14T13:41:37.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:37 smithi008 ceph-mon[67070]: pgmap v28: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.0 MiB/s wr, 488 op/s 2024-04-14T13:41:38.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:37 smithi012 ceph-mon[53289]: pgmap v28: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.0 MiB/s wr, 488 op/s 2024-04-14T13:41:38.342 INFO:teuthology.orchestra.run.smithi008.stdout: 5 179612 11.76 MB/sec execute 252 sec latency 35.121 ms 2024-04-14T13:41:39.069 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:38 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:39.069 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:38 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:39.069 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:38 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.wqbvfi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-14T13:41:39.070 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:38 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:39.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:38 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:39.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:38 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:39.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:38 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.wqbvfi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-14T13:41:39.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:38 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:39.342 INFO:teuthology.orchestra.run.smithi008.stdout: 5 180027 11.76 MB/sec execute 253 sec latency 45.965 ms 2024-04-14T13:41:40.075 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:39 smithi008 ceph-mon[67070]: Reconfiguring mds.foofs.smithi008.wqbvfi (monmap changed)... 2024-04-14T13:41:40.075 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:39 smithi008 ceph-mon[67070]: Reconfiguring daemon mds.foofs.smithi008.wqbvfi on smithi008 2024-04-14T13:41:40.075 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:39 smithi008 ceph-mon[67070]: pgmap v29: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.0 MiB/s wr, 488 op/s 2024-04-14T13:41:40.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:39 smithi012 ceph-mon[53289]: Reconfiguring mds.foofs.smithi008.wqbvfi (monmap changed)... 2024-04-14T13:41:40.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:39 smithi012 ceph-mon[53289]: Reconfiguring daemon mds.foofs.smithi008.wqbvfi on smithi008 2024-04-14T13:41:40.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:39 smithi012 ceph-mon[53289]: pgmap v29: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.0 MiB/s wr, 488 op/s 2024-04-14T13:41:40.342 INFO:teuthology.orchestra.run.smithi008.stdout: 5 180545 11.77 MB/sec execute 254 sec latency 24.464 ms 2024-04-14T13:41:41.345 INFO:teuthology.orchestra.run.smithi008.stdout: 5 181094 11.81 MB/sec execute 255 sec latency 29.838 ms 2024-04-14T13:41:41.846 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:41 smithi008 ceph-mon[67070]: pgmap v30: 97 pgs: 97 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 742 op/s 2024-04-14T13:41:41.846 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:41 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:41.847 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:41 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:41.847 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:41 smithi008 ceph-mon[67070]: Reconfiguring crash.smithi012 (monmap changed)... 2024-04-14T13:41:41.847 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:41 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi012", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-14T13:41:41.847 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:41 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:41.847 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:41 smithi008 ceph-mon[67070]: Reconfiguring daemon crash.smithi012 on smithi012 2024-04-14T13:41:42.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:41 smithi012 ceph-mon[53289]: pgmap v30: 97 pgs: 97 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 742 op/s 2024-04-14T13:41:42.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:41 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:42.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:41 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:42.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:41 smithi012 ceph-mon[53289]: Reconfiguring crash.smithi012 (monmap changed)... 2024-04-14T13:41:42.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:41 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi012", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-14T13:41:42.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:41 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:42.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:41 smithi012 ceph-mon[53289]: Reconfiguring daemon crash.smithi012 on smithi012 2024-04-14T13:41:42.347 INFO:teuthology.orchestra.run.smithi008.stdout: 5 181815 11.83 MB/sec execute 256 sec latency 27.001 ms 2024-04-14T13:41:43.347 INFO:teuthology.orchestra.run.smithi008.stdout: 5 182501 11.84 MB/sec execute 257 sec latency 32.829 ms 2024-04-14T13:41:43.689 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:43 smithi012 ceph-mon[53289]: pgmap v31: 97 pgs: 97 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.0 MiB/s wr, 506 op/s 2024-04-14T13:41:43.689 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:43 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:43.689 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:43 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:43.690 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:43 smithi012 ceph-mon[53289]: Reconfiguring mgr.smithi012.ksvnxm (monmap changed)... 2024-04-14T13:41:43.690 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:43 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi012.ksvnxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:41:43.690 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:43 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:41:43.690 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:43 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:43.690 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:43 smithi012 ceph-mon[53289]: Reconfiguring daemon mgr.smithi012.ksvnxm on smithi012 2024-04-14T13:41:43.856 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:43 smithi008 ceph-mon[67070]: pgmap v31: 97 pgs: 97 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.0 MiB/s wr, 506 op/s 2024-04-14T13:41:43.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:43 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:43.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:43 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:43.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:43 smithi008 ceph-mon[67070]: Reconfiguring mgr.smithi012.ksvnxm (monmap changed)... 2024-04-14T13:41:43.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:43 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi012.ksvnxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:41:43.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:43 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:41:43.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:43 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:43.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:43 smithi008 ceph-mon[67070]: Reconfiguring daemon mgr.smithi012.ksvnxm on smithi012 2024-04-14T13:41:44.347 INFO:teuthology.orchestra.run.smithi008.stdout: 5 182831 11.83 MB/sec execute 258 sec latency 43.164 ms 2024-04-14T13:41:45.347 INFO:teuthology.orchestra.run.smithi008.stdout: 5 183095 11.80 MB/sec execute 259 sec latency 62.419 ms 2024-04-14T13:41:45.897 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:45 smithi008 ceph-mon[67070]: pgmap v32: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 838 op/s 2024-04-14T13:41:45.898 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:45 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:45.898 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:45 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:45.898 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:45 smithi008 ceph-mon[67070]: Reconfiguring mon.smithi012 (monmap changed)... 2024-04-14T13:41:45.898 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:45 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:41:45.898 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:45 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:41:45.898 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:45 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:45.898 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:45 smithi008 ceph-mon[67070]: Reconfiguring daemon mon.smithi012 on smithi012 2024-04-14T13:41:46.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:45 smithi012 ceph-mon[53289]: pgmap v32: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 838 op/s 2024-04-14T13:41:46.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:45 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:46.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:45 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:46.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:45 smithi012 ceph-mon[53289]: Reconfiguring mon.smithi012 (monmap changed)... 2024-04-14T13:41:46.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:45 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:41:46.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:45 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:41:46.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:45 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:46.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:45 smithi012 ceph-mon[53289]: Reconfiguring daemon mon.smithi012 on smithi012 2024-04-14T13:41:46.347 INFO:teuthology.orchestra.run.smithi008.stdout: 5 183400 11.80 MB/sec execute 260 sec latency 63.882 ms 2024-04-14T13:41:47.347 INFO:teuthology.orchestra.run.smithi008.stdout: 5 183818 11.78 MB/sec execute 261 sec latency 57.546 ms 2024-04-14T13:41:47.680 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:47 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:47.681 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:47 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:47.681 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:47 smithi008 ceph-mon[67070]: pgmap v33: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.8 MiB/s wr, 586 op/s 2024-04-14T13:41:47.681 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:47 smithi008 ceph-mon[67070]: Reconfiguring osd.4 (monmap changed)... 2024-04-14T13:41:47.681 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:47 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T13:41:47.681 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:47 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:47.681 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:47 smithi008 ceph-mon[67070]: Reconfiguring daemon osd.4 on smithi012 2024-04-14T13:41:47.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:47 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:47.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:47 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:47.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:47 smithi012 ceph-mon[53289]: pgmap v33: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.8 MiB/s wr, 586 op/s 2024-04-14T13:41:47.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:47 smithi012 ceph-mon[53289]: Reconfiguring osd.4 (monmap changed)... 2024-04-14T13:41:47.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:47 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T13:41:47.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:47 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:47.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:47 smithi012 ceph-mon[53289]: Reconfiguring daemon osd.4 on smithi012 2024-04-14T13:41:48.347 INFO:teuthology.orchestra.run.smithi008.stdout: 5 184300 11.79 MB/sec execute 262 sec latency 41.943 ms 2024-04-14T13:41:48.686 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:41:48.686 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:48.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:41:48.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:49.347 INFO:teuthology.orchestra.run.smithi008.stdout: 5 184933 11.83 MB/sec execute 263 sec latency 34.737 ms 2024-04-14T13:41:49.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:49 smithi012 ceph-mon[53289]: pgmap v34: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.8 MiB/s wr, 586 op/s 2024-04-14T13:41:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:49 smithi012 ceph-mon[53289]: Reconfiguring osd.5 (monmap changed)... 2024-04-14T13:41:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T13:41:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:49 smithi012 ceph-mon[53289]: Reconfiguring daemon osd.5 on smithi012 2024-04-14T13:41:49.699 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:49.700 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:49 smithi008 ceph-mon[67070]: pgmap v34: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.8 MiB/s wr, 586 op/s 2024-04-14T13:41:49.700 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:49 smithi008 ceph-mon[67070]: Reconfiguring osd.5 (monmap changed)... 2024-04-14T13:41:49.700 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T13:41:49.700 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:49.700 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:49 smithi008 ceph-mon[67070]: Reconfiguring daemon osd.5 on smithi012 2024-04-14T13:41:50.348 INFO:teuthology.orchestra.run.smithi008.stdout: 5 185268 11.82 MB/sec execute 264 sec latency 39.975 ms 2024-04-14T13:41:51.348 INFO:teuthology.orchestra.run.smithi008.stdout: 5 185488 11.79 MB/sec execute 265 sec latency 50.779 ms 2024-04-14T13:41:51.746 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:51 smithi008 ceph-mon[67070]: pgmap v35: 97 pgs: 97 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 840 op/s 2024-04-14T13:41:51.747 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:51 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:51.747 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:51 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:51.747 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:51 smithi008 ceph-mon[67070]: Reconfiguring osd.6 (monmap changed)... 2024-04-14T13:41:51.747 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:51 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T13:41:51.747 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:51 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:51.747 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:51 smithi008 ceph-mon[67070]: Reconfiguring daemon osd.6 on smithi012 2024-04-14T13:41:52.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:51 smithi012 ceph-mon[53289]: pgmap v35: 97 pgs: 97 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 840 op/s 2024-04-14T13:41:52.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:51 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:52.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:51 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:52.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:51 smithi012 ceph-mon[53289]: Reconfiguring osd.6 (monmap changed)... 2024-04-14T13:41:52.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:51 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T13:41:52.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:51 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:52.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:51 smithi012 ceph-mon[53289]: Reconfiguring daemon osd.6 on smithi012 2024-04-14T13:41:52.348 INFO:teuthology.orchestra.run.smithi008.stdout: 5 185800 11.79 MB/sec execute 266 sec latency 45.090 ms 2024-04-14T13:41:53.348 INFO:teuthology.orchestra.run.smithi008.stdout: 5 186409 11.79 MB/sec execute 267 sec latency 28.641 ms 2024-04-14T13:41:53.918 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:53 smithi012 ceph-mon[53289]: pgmap v36: 97 pgs: 97 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.2 MiB/s wr, 585 op/s 2024-04-14T13:41:53.918 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:53 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:53.918 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:53 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:53.918 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:53 smithi012 ceph-mon[53289]: Reconfiguring osd.7 (monmap changed)... 2024-04-14T13:41:53.918 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:53 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T13:41:53.918 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:53 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:53.918 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:53 smithi012 ceph-mon[53289]: Reconfiguring daemon osd.7 on smithi012 2024-04-14T13:41:54.021 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:53 smithi008 ceph-mon[67070]: pgmap v36: 97 pgs: 97 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.2 MiB/s wr, 585 op/s 2024-04-14T13:41:54.022 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:53 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:54.022 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:53 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:54.022 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:53 smithi008 ceph-mon[67070]: Reconfiguring osd.7 (monmap changed)... 2024-04-14T13:41:54.022 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:53 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T13:41:54.022 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:53 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:54.022 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:53 smithi008 ceph-mon[67070]: Reconfiguring daemon osd.7 on smithi012 2024-04-14T13:41:54.348 INFO:teuthology.orchestra.run.smithi008.stdout: 5 187065 11.81 MB/sec execute 268 sec latency 33.656 ms 2024-04-14T13:41:55.349 INFO:teuthology.orchestra.run.smithi008.stdout: 5 187650 11.82 MB/sec execute 269 sec latency 97.176 ms 2024-04-14T13:41:56.056 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:55 smithi008 ceph-mon[67070]: pgmap v37: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 840 op/s 2024-04-14T13:41:56.056 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:55 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:56.057 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:55 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:56.057 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:55 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi012.vjovzp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-14T13:41:56.057 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:55 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:56.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:55 smithi012 ceph-mon[53289]: pgmap v37: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 840 op/s 2024-04-14T13:41:56.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:55 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:56.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:55 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:56.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:55 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi012.vjovzp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-14T13:41:56.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:55 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:41:56.348 INFO:teuthology.orchestra.run.smithi008.stdout: 5 188191 11.86 MB/sec execute 270 sec latency 24.171 ms 2024-04-14T13:41:57.066 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:56 smithi008 ceph-mon[67070]: Reconfiguring mds.foofs.smithi012.vjovzp (monmap changed)... 2024-04-14T13:41:57.067 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:56 smithi008 ceph-mon[67070]: Reconfiguring daemon mds.foofs.smithi012.vjovzp on smithi012 2024-04-14T13:41:57.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:56 smithi012 ceph-mon[53289]: Reconfiguring mds.foofs.smithi012.vjovzp (monmap changed)... 2024-04-14T13:41:57.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:56 smithi012 ceph-mon[53289]: Reconfiguring daemon mds.foofs.smithi012.vjovzp on smithi012 2024-04-14T13:41:57.348 INFO:teuthology.orchestra.run.smithi008.stdout: 5 188731 11.86 MB/sec execute 271 sec latency 53.158 ms 2024-04-14T13:41:58.074 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:57 smithi008 ceph-mon[67070]: pgmap v38: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.1 MiB/s wr, 508 op/s 2024-04-14T13:41:58.074 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:57 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:58.074 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:57 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:58.074 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:57 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:41:58.074 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:57 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:41:58.074 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:57 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:41:58.074 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:57 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:58.074 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:57 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi008"}]: dispatch 2024-04-14T13:41:58.074 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:57 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi008"}]': finished 2024-04-14T13:41:58.074 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:57 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi012"}]: dispatch 2024-04-14T13:41:58.074 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:57 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi012"}]': finished 2024-04-14T13:41:58.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:57 smithi012 ceph-mon[53289]: pgmap v38: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.1 MiB/s wr, 508 op/s 2024-04-14T13:41:58.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:57 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:58.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:57 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:58.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:57 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:41:58.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:57 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:41:58.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:57 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:41:58.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:57 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:41:58.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:57 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi008"}]: dispatch 2024-04-14T13:41:58.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:57 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi008"}]': finished 2024-04-14T13:41:58.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:57 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi012"}]: dispatch 2024-04-14T13:41:58.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:57 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi012"}]': finished 2024-04-14T13:41:58.348 INFO:teuthology.orchestra.run.smithi008.stdout: 5 189050 11.84 MB/sec execute 272 sec latency 52.266 ms 2024-04-14T13:41:59.095 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:58 smithi008 ceph-mon[67070]: Upgrade: Setting container_image for all mon 2024-04-14T13:41:59.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:58 smithi012 ceph-mon[53289]: Upgrade: Setting container_image for all mon 2024-04-14T13:41:59.348 INFO:teuthology.orchestra.run.smithi008.stdout: 5 189346 11.83 MB/sec execute 273 sec latency 69.237 ms 2024-04-14T13:42:00.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:41:59 smithi008 ceph-mon[67070]: pgmap v39: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.1 MiB/s wr, 508 op/s 2024-04-14T13:42:00.257 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:41:59 smithi012 ceph-mon[53289]: pgmap v39: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.1 MiB/s wr, 508 op/s 2024-04-14T13:42:00.348 INFO:teuthology.orchestra.run.smithi008.stdout: 5 189648 11.80 MB/sec execute 274 sec latency 59.659 ms 2024-04-14T13:42:01.348 INFO:teuthology.orchestra.run.smithi008.stdout: 5 189920 11.79 MB/sec execute 275 sec latency 67.857 ms 2024-04-14T13:42:02.154 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:01 smithi008 ceph-mon[67070]: pgmap v40: 97 pgs: 97 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 809 op/s 2024-04-14T13:42:02.155 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:01 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:02.155 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:01 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-14T13:42:02.155 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:01 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:42:02.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:01 smithi012 ceph-mon[53289]: pgmap v40: 97 pgs: 97 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 809 op/s 2024-04-14T13:42:02.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:01 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:02.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:01 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-14T13:42:02.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:01 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:42:02.348 INFO:teuthology.orchestra.run.smithi008.stdout: 5 190129 11.77 MB/sec execute 276 sec latency 62.107 ms 2024-04-14T13:42:02.794 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:42:03.176 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:02 smithi008 ceph-mon[67070]: Upgrade: Updating crash.smithi008 (1/2) 2024-04-14T13:42:03.176 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:02 smithi008 ceph-mon[67070]: Deploying daemon crash.smithi008 on smithi008 2024-04-14T13:42:03.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:02 smithi012 ceph-mon[53289]: Upgrade: Updating crash.smithi008 (1/2) 2024-04-14T13:42:03.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:02 smithi012 ceph-mon[53289]: Deploying daemon crash.smithi008 on smithi008 2024-04-14T13:42:03.348 INFO:teuthology.orchestra.run.smithi008.stdout: 5 190441 11.77 MB/sec execute 277 sec latency 46.553 ms 2024-04-14T13:42:03.891 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:42:03.891 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (3m) 66s ago 13m 16.8M - 0.23.0 ba2b418f427c cfddb9a7f858 2024-04-14T13:42:03.892 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (13m) 66s ago 13m 6405k - 17.2.0 e1d6a67b021e 68b810a9ff93 2024-04-14T13:42:03.892 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (12m) 50s ago 12m 6400k - 17.2.0 e1d6a67b021e 6bc31c51b938 2024-04-14T13:42:03.892 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 66s ago 12m - - 2024-04-14T13:42:03.892 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (7m) 66s ago 7m 38.0M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:42:03.892 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (7m) 50s ago 7m 14.2M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:42:03.892 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (2m) 66s ago 14m 477M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:42:03.892 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (108s) 50s ago 12m 449M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:42:03.892 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (87s) 66s ago 14m 30.4M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 673dd6f4c8d8 2024-04-14T13:42:03.892 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (57s) 50s ago 12m 24.4M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 d498dec29c4f 2024-04-14T13:42:03.892 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (6m) 66s ago 6m 275M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:42:03.892 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (6m) 50s ago 6m 88.4M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:42:03.892 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (3m) 66s ago 12m 14.9M - 1.3.1 1dbe0e931976 646dfca70be4 2024-04-14T13:42:03.892 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (3m) 50s ago 11m 18.4M - 1.3.1 1dbe0e931976 0a60dcd4044c 2024-04-14T13:42:03.892 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (11m) 66s ago 11m 323M 1937M 17.2.0 e1d6a67b021e 3b8cdea7890c 2024-04-14T13:42:03.892 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (11m) 66s ago 11m 488M 1937M 17.2.0 e1d6a67b021e be86642c7790 2024-04-14T13:42:03.893 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (10m) 66s ago 10m 344M 1937M 17.2.0 e1d6a67b021e b1017f00578f 2024-04-14T13:42:03.893 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (10m) 66s ago 10m 510M 1937M 17.2.0 e1d6a67b021e 66fca4b191be 2024-04-14T13:42:03.893 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (9m) 50s ago 9m 570M 2705M 17.2.0 e1d6a67b021e 43b02cb64951 2024-04-14T13:42:03.893 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (9m) 50s ago 9m 362M 2705M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:42:03.893 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (8m) 50s ago 8m 555M 2705M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:42:03.893 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (8m) 50s ago 8m 463M 2705M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:42:03.893 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (115s) 66s ago 11m 64.3M - 2.33.4 514e6a882f6e 12f97f75d620 2024-04-14T13:42:03.935 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:03 smithi008 ceph-mon[67070]: pgmap v41: 97 pgs: 97 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.6 MiB/s wr, 556 op/s 2024-04-14T13:42:03.936 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:03 smithi008 ceph-mon[67070]: from='client.44121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:42:03.936 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:42:03.936 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:03 smithi008 ceph-mon[67070]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:42:03.936 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:03 smithi008 ceph-mon[67070]: from='client.34184 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:42:04.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:03 smithi012 ceph-mon[53289]: pgmap v41: 97 pgs: 97 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.6 MiB/s wr, 556 op/s 2024-04-14T13:42:04.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:03 smithi012 ceph-mon[53289]: from='client.44121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:42:04.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:42:04.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:03 smithi012 ceph-mon[53289]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:42:04.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:03 smithi012 ceph-mon[53289]: from='client.34184 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:42:04.348 INFO:teuthology.orchestra.run.smithi008.stdout: 5 190670 11.73 MB/sec execute 278 sec latency 75.441 ms 2024-04-14T13:42:04.580 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:42:04.580 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:42:04.580 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:42:04.580 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:42:04.580 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:42:04.580 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:42:04.581 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:42:04.581 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:42:04.581 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T13:42:04.581 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:42:04.581 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:42:04.581 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:42:04.581 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:42:04.581 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:42:04.581 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-14T13:42:04.581 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 4 2024-04-14T13:42:04.581 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:42:04.581 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:42:05.146 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:42:05.146 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:42:05.146 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:42:05.146 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:42:05.146 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-14T13:42:05.146 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-04-14T13:42:05.146 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-04-14T13:42:05.146 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:42:05.146 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "4/23 daemons upgraded", 2024-04-14T13:42:05.147 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading crash daemons", 2024-04-14T13:42:05.147 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:42:05.147 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:42:05.178 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:04 smithi008 ceph-mon[67070]: pgmap v42: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 721 op/s 2024-04-14T13:42:05.178 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:04 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/3955406375' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:42:05.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:04 smithi012 ceph-mon[53289]: pgmap v42: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 721 op/s 2024-04-14T13:42:05.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:04 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/3955406375' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:42:05.348 INFO:teuthology.orchestra.run.smithi008.stdout: 5 190990 11.72 MB/sec execute 279 sec latency 42.378 ms 2024-04-14T13:42:05.823 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T13:42:05.823 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:42:05.823 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-14T13:42:05.984 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:05 smithi008 ceph-mon[67070]: from='client.34190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:42:05.984 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:05 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/3170495293' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:42:06.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:05 smithi012 ceph-mon[53289]: from='client.34190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:42:06.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:05 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/3170495293' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:42:06.348 INFO:teuthology.orchestra.run.smithi008.stdout: 5 191216 11.70 MB/sec execute 280 sec latency 48.887 ms 2024-04-14T13:42:07.016 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:06 smithi008 ceph-mon[67070]: pgmap v43: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.6 MiB/s wr, 466 op/s 2024-04-14T13:42:07.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:06 smithi012 ceph-mon[53289]: pgmap v43: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.6 MiB/s wr, 466 op/s 2024-04-14T13:42:07.348 INFO:teuthology.orchestra.run.smithi008.stdout: 5 191406 11.68 MB/sec execute 281 sec latency 75.526 ms 2024-04-14T13:42:08.349 INFO:teuthology.orchestra.run.smithi008.stdout: 5 191896 11.70 MB/sec execute 282 sec latency 32.532 ms 2024-04-14T13:42:08.526 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:08 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:08.526 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:08 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:08.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:08 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:08.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:08 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:09.349 INFO:teuthology.orchestra.run.smithi008.stdout: 5 192361 11.70 MB/sec execute 283 sec latency 43.827 ms 2024-04-14T13:42:09.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:09 smithi008 ceph-mon[67070]: pgmap v44: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.6 MiB/s wr, 466 op/s 2024-04-14T13:42:09.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:09 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:09.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:09 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi012", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-14T13:42:09.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:09 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:42:09.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:09 smithi012 ceph-mon[53289]: pgmap v44: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.6 MiB/s wr, 466 op/s 2024-04-14T13:42:09.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:09 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:09.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:09 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi012", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-14T13:42:09.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:09 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:42:10.349 INFO:teuthology.orchestra.run.smithi008.stdout: 5 192657 11.69 MB/sec execute 284 sec latency 43.590 ms 2024-04-14T13:42:10.547 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:10 smithi008 ceph-mon[67070]: Upgrade: Updating crash.smithi012 (2/2) 2024-04-14T13:42:10.547 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:10 smithi008 ceph-mon[67070]: Deploying daemon crash.smithi012 on smithi012 2024-04-14T13:42:10.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:10 smithi012 ceph-mon[53289]: Upgrade: Updating crash.smithi012 (2/2) 2024-04-14T13:42:10.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:10 smithi012 ceph-mon[53289]: Deploying daemon crash.smithi012 on smithi012 2024-04-14T13:42:11.349 INFO:teuthology.orchestra.run.smithi008.stdout: 5 192960 11.67 MB/sec execute 285 sec latency 55.309 ms 2024-04-14T13:42:11.574 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:11 smithi008 ceph-mon[67070]: pgmap v45: 97 pgs: 97 active+clean; 342 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.7 MiB/s wr, 659 op/s 2024-04-14T13:42:11.657 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:11 smithi012 ceph-mon[53289]: pgmap v45: 97 pgs: 97 active+clean; 342 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.7 MiB/s wr, 659 op/s 2024-04-14T13:42:12.349 INFO:teuthology.orchestra.run.smithi008.stdout: 5 193284 11.66 MB/sec execute 286 sec latency 48.170 ms 2024-04-14T13:42:13.349 INFO:teuthology.orchestra.run.smithi008.stdout: 5 193522 11.64 MB/sec execute 287 sec latency 60.075 ms 2024-04-14T13:42:13.633 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:13 smithi008 ceph-mon[67070]: pgmap v46: 97 pgs: 97 active+clean; 342 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.5 MiB/s wr, 357 op/s 2024-04-14T13:42:13.634 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:13.634 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:13.634 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:42:14.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:13 smithi012 ceph-mon[53289]: pgmap v46: 97 pgs: 97 active+clean; 342 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.5 MiB/s wr, 357 op/s 2024-04-14T13:42:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:42:14.349 INFO:teuthology.orchestra.run.smithi008.stdout: 5 193903 11.64 MB/sec execute 288 sec latency 39.712 ms 2024-04-14T13:42:15.350 INFO:teuthology.orchestra.run.smithi008.stdout: 5 194290 11.62 MB/sec execute 289 sec latency 47.857 ms 2024-04-14T13:42:15.649 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:15 smithi008 ceph-mon[67070]: pgmap v47: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.4 MiB/s wr, 560 op/s 2024-04-14T13:42:15.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:15 smithi012 ceph-mon[53289]: pgmap v47: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.4 MiB/s wr, 560 op/s 2024-04-14T13:42:16.349 INFO:teuthology.orchestra.run.smithi008.stdout: 5 194726 11.63 MB/sec execute 290 sec latency 37.094 ms 2024-04-14T13:42:17.349 INFO:teuthology.orchestra.run.smithi008.stdout: 5 195109 11.63 MB/sec execute 291 sec latency 46.638 ms 2024-04-14T13:42:17.689 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:17 smithi008 ceph-mon[67070]: pgmap v48: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.0 MiB/s wr, 395 op/s 2024-04-14T13:42:17.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:17 smithi012 ceph-mon[53289]: pgmap v48: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.0 MiB/s wr, 395 op/s 2024-04-14T13:42:18.349 INFO:teuthology.orchestra.run.smithi008.stdout: 5 195578 11.63 MB/sec execute 292 sec latency 43.859 ms 2024-04-14T13:42:19.098 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:18 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:19.098 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:18 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:19.098 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:18 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:42:19.099 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:18 smithi008 ceph-mon[67070]: pgmap v49: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.0 MiB/s wr, 395 op/s 2024-04-14T13:42:19.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:18 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:19.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:18 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:19.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:18 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:42:19.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:18 smithi012 ceph-mon[53289]: pgmap v49: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.0 MiB/s wr, 395 op/s 2024-04-14T13:42:19.349 INFO:teuthology.orchestra.run.smithi008.stdout: 5 196011 11.64 MB/sec execute 293 sec latency 35.600 ms 2024-04-14T13:42:20.349 INFO:teuthology.orchestra.run.smithi008.stdout: 5 196390 11.63 MB/sec execute 294 sec latency 52.119 ms 2024-04-14T13:42:20.471 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:20 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:20.471 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:20 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:20.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:20 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:20.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:20 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:21.350 INFO:teuthology.orchestra.run.smithi008.stdout: 5 196818 11.63 MB/sec execute 295 sec latency 67.627 ms 2024-04-14T13:42:21.484 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:21 smithi008 ceph-mon[67070]: mgrmap e42: smithi008.yemlhr(active, since 92s), standbys: smithi012.ksvnxm 2024-04-14T13:42:21.484 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:21 smithi008 ceph-mon[67070]: pgmap v50: 97 pgs: 97 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.7 MiB/s wr, 646 op/s 2024-04-14T13:42:21.696 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:21 smithi012 ceph-mon[53289]: mgrmap e42: smithi008.yemlhr(active, since 92s), standbys: smithi012.ksvnxm 2024-04-14T13:42:21.697 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:21 smithi012 ceph-mon[53289]: pgmap v50: 97 pgs: 97 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.7 MiB/s wr, 646 op/s 2024-04-14T13:42:22.350 INFO:teuthology.orchestra.run.smithi008.stdout: 5 197115 11.61 MB/sec execute 296 sec latency 74.265 ms 2024-04-14T13:42:23.300 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:23 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:23.301 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:23 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:23.301 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:23 smithi008 ceph-mon[67070]: pgmap v51: 97 pgs: 97 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.6 MiB/s wr, 454 op/s 2024-04-14T13:42:23.301 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:23 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:23.301 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:23 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:23.350 INFO:teuthology.orchestra.run.smithi008.stdout: 5 197445 11.61 MB/sec execute 297 sec latency 49.291 ms 2024-04-14T13:42:23.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:23 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:23.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:23 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:23.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:23 smithi012 ceph-mon[53289]: pgmap v51: 97 pgs: 97 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.6 MiB/s wr, 454 op/s 2024-04-14T13:42:23.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:23 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:23.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:23 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:24.350 INFO:teuthology.orchestra.run.smithi008.stdout: 5 197750 11.59 MB/sec execute 298 sec latency 47.995 ms 2024-04-14T13:42:25.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:24 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:25.073 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:24 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:25.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:24 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:25.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:24 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:25.350 INFO:teuthology.orchestra.run.smithi008.stdout: 5 198147 11.59 MB/sec execute 299 sec latency 43.101 ms 2024-04-14T13:42:26.099 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:25 smithi008 ceph-mon[67070]: pgmap v52: 97 pgs: 97 active+clean; 341 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.7 MiB/s wr, 679 op/s 2024-04-14T13:42:26.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:25 smithi012 ceph-mon[53289]: pgmap v52: 97 pgs: 97 active+clean; 341 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.7 MiB/s wr, 679 op/s 2024-04-14T13:42:26.350 INFO:teuthology.orchestra.run.smithi008.stdout: 5 198386 11.57 MB/sec execute 300 sec latency 44.387 ms 2024-04-14T13:42:27.350 INFO:teuthology.orchestra.run.smithi008.stdout: 5 198649 11.56 MB/sec execute 301 sec latency 46.937 ms 2024-04-14T13:42:27.905 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:27 smithi008 ceph-mon[67070]: pgmap v53: 97 pgs: 97 active+clean; 341 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 476 op/s 2024-04-14T13:42:28.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:27 smithi012 ceph-mon[53289]: pgmap v53: 97 pgs: 97 active+clean; 341 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 476 op/s 2024-04-14T13:42:28.350 INFO:teuthology.orchestra.run.smithi008.stdout: 5 199025 11.56 MB/sec execute 302 sec latency 39.762 ms 2024-04-14T13:42:29.350 INFO:teuthology.orchestra.run.smithi008.stdout: 5 199379 11.55 MB/sec execute 303 sec latency 60.639 ms 2024-04-14T13:42:29.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:29 smithi008 ceph-mon[67070]: pgmap v54: 97 pgs: 97 active+clean; 341 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 476 op/s 2024-04-14T13:42:29.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:29.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:29.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:42:29.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:42:29.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:29.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:29.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:42:29.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:42:29.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:42:29.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:42:29.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:29.969 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi008"}]: dispatch 2024-04-14T13:42:29.969 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi008"}]': finished 2024-04-14T13:42:29.969 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi012"}]: dispatch 2024-04-14T13:42:29.969 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi012"}]': finished 2024-04-14T13:42:29.969 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-14T13:42:30.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:29 smithi012 ceph-mon[53289]: pgmap v54: 97 pgs: 97 active+clean; 341 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 476 op/s 2024-04-14T13:42:30.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:30.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:30.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:42:30.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:42:30.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:30.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:30.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:42:30.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:42:30.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:42:30.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:42:30.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:30.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi008"}]: dispatch 2024-04-14T13:42:30.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi008"}]': finished 2024-04-14T13:42:30.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi012"}]: dispatch 2024-04-14T13:42:30.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi012"}]': finished 2024-04-14T13:42:30.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-14T13:42:30.350 INFO:teuthology.orchestra.run.smithi008.stdout: 5 199598 11.53 MB/sec execute 304 sec latency 70.273 ms 2024-04-14T13:42:30.984 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:30 smithi008 ceph-mon[67070]: pgmap v55: 97 pgs: 97 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.7 MiB/s wr, 757 op/s 2024-04-14T13:42:30.984 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:30 smithi008 ceph-mon[67070]: pgmap v56: 97 pgs: 97 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.8 MiB/s wr, 583 op/s 2024-04-14T13:42:30.984 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:30 smithi008 ceph-mon[67070]: Upgrade: Setting container_image for all crash 2024-04-14T13:42:30.984 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:30 smithi008 ceph-mon[67070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-14T13:42:30.984 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:30 smithi008 ceph-mon[67070]: Upgrade: osd.0 is safe to restart 2024-04-14T13:42:31.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:30 smithi012 ceph-mon[53289]: pgmap v55: 97 pgs: 97 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.7 MiB/s wr, 757 op/s 2024-04-14T13:42:31.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:30 smithi012 ceph-mon[53289]: pgmap v56: 97 pgs: 97 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.8 MiB/s wr, 583 op/s 2024-04-14T13:42:31.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:30 smithi012 ceph-mon[53289]: Upgrade: Setting container_image for all crash 2024-04-14T13:42:31.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:30 smithi012 ceph-mon[53289]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-14T13:42:31.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:30 smithi012 ceph-mon[53289]: Upgrade: osd.0 is safe to restart 2024-04-14T13:42:31.350 INFO:teuthology.orchestra.run.smithi008.stdout: 5 199857 11.52 MB/sec execute 305 sec latency 58.030 ms 2024-04-14T13:42:32.350 INFO:teuthology.orchestra.run.smithi008.stdout: 5 200131 11.50 MB/sec execute 306 sec latency 61.997 ms 2024-04-14T13:42:32.526 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:32 smithi008 ceph-mon[67070]: pgmap v57: 97 pgs: 97 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.8 MiB/s wr, 583 op/s 2024-04-14T13:42:32.526 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:32 smithi008 ceph-mon[67070]: Upgrade: Updating osd.0 2024-04-14T13:42:32.526 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:32 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:32.527 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:32 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T13:42:32.527 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:32 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:42:32.527 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:32 smithi008 ceph-mon[67070]: Deploying daemon osd.0 on smithi008 2024-04-14T13:42:32.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:32 smithi012 ceph-mon[53289]: pgmap v57: 97 pgs: 97 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.8 MiB/s wr, 583 op/s 2024-04-14T13:42:32.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:32 smithi012 ceph-mon[53289]: Upgrade: Updating osd.0 2024-04-14T13:42:32.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:32 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:32.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:32 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T13:42:32.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:32 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:42:32.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:32 smithi012 ceph-mon[53289]: Deploying daemon osd.0 on smithi008 2024-04-14T13:42:33.350 INFO:teuthology.orchestra.run.smithi008.stdout: 5 200569 11.50 MB/sec execute 307 sec latency 56.864 ms 2024-04-14T13:42:33.439 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:33 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:42:33.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:33 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:42:34.350 INFO:teuthology.orchestra.run.smithi008.stdout: 5 201207 11.52 MB/sec execute 308 sec latency 62.302 ms 2024-04-14T13:42:34.571 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:34 smithi008 ceph-mon[67070]: pgmap v58: 97 pgs: 97 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.9 MiB/s wr, 268 op/s 2024-04-14T13:42:34.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:34 smithi012 ceph-mon[53289]: pgmap v58: 97 pgs: 97 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.9 MiB/s wr, 268 op/s 2024-04-14T13:42:35.350 INFO:teuthology.orchestra.run.smithi008.stdout: 5 201865 11.53 MB/sec execute 309 sec latency 31.733 ms 2024-04-14T13:42:36.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 202590 11.57 MB/sec execute 310 sec latency 34.306 ms 2024-04-14T13:42:36.461 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:42:36.552 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:36 smithi008 ceph-mon[67070]: pgmap v59: 97 pgs: 97 active+clean; 346 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.9 MiB/s wr, 568 op/s 2024-04-14T13:42:36.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:36 smithi012 ceph-mon[53289]: pgmap v59: 97 pgs: 97 active+clean; 346 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.9 MiB/s wr, 568 op/s 2024-04-14T13:42:37.312 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:37 smithi008 systemd[1]: Stopping Ceph osd.0 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:42:37.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 203007 11.57 MB/sec execute 311 sec latency 49.876 ms 2024-04-14T13:42:37.567 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:37 smithi008 ceph-mon[67070]: from='client.44143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:42:37.596 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:42:37.596 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (4m) 15s ago 13m 15.2M - 0.23.0 ba2b418f427c cfddb9a7f858 2024-04-14T13:42:37.596 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (30s) 15s ago 13m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 1e69eabdcc36 2024-04-14T13:42:37.596 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (24s) 19s ago 12m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 60c7a2992eb0 2024-04-14T13:42:37.596 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 15s ago 13m - - 2024-04-14T13:42:37.596 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (7m) 15s ago 7m 38.5M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:42:37.596 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (7m) 19s ago 7m 14.4M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:42:37.596 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (3m) 15s ago 14m 501M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:42:37.597 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (2m) 19s ago 12m 451M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:42:37.597 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (2m) 15s ago 15m 41.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 673dd6f4c8d8 2024-04-14T13:42:37.597 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (90s) 19s ago 12m 35.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 d498dec29c4f 2024-04-14T13:42:37.597 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (7m) 15s ago 7m 285M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:42:37.597 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (7m) 19s ago 7m 88.5M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:42:37.597 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (4m) 15s ago 13m 14.1M - 1.3.1 1dbe0e931976 646dfca70be4 2024-04-14T13:42:37.597 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (3m) 19s ago 12m 18.4M - 1.3.1 1dbe0e931976 0a60dcd4044c 2024-04-14T13:42:37.597 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (12m) 15s ago 12m 411M 1937M 17.2.0 e1d6a67b021e 3b8cdea7890c 2024-04-14T13:42:37.597 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (11m) 15s ago 11m 590M 1937M 17.2.0 e1d6a67b021e be86642c7790 2024-04-14T13:42:37.597 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (11m) 15s ago 11m 464M 1937M 17.2.0 e1d6a67b021e b1017f00578f 2024-04-14T13:42:37.597 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (10m) 15s ago 10m 601M 1937M 17.2.0 e1d6a67b021e 66fca4b191be 2024-04-14T13:42:37.597 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (10m) 19s ago 10m 614M 2705M 17.2.0 e1d6a67b021e 43b02cb64951 2024-04-14T13:42:37.597 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (9m) 19s ago 9m 430M 2705M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:42:37.597 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (9m) 19s ago 9m 341M 2705M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:42:37.597 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (9m) 19s ago 9m 552M 2705M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:42:37.598 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (2m) 15s ago 12m 65.0M - 2.33.4 514e6a882f6e 12f97f75d620 2024-04-14T13:42:37.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:37 smithi012 ceph-mon[53289]: from='client.44143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:42:37.826 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:37 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0[34981]: 2024-04-14T13:42:37.604+0000 7fa45db8a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:42:37.826 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:37 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0[34981]: 2024-04-14T13:42:37.604+0000 7fa45db8a700 -1 osd.0 73 *** Got signal Terminated *** 2024-04-14T13:42:37.826 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:37 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0[34981]: 2024-04-14T13:42:37.604+0000 7fa45db8a700 -1 osd.0 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T13:42:38.263 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:42:38.263 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:42:38.263 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:42:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:42:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:42:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:42:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:42:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:42:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T13:42:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:42:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:42:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:42:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:42:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:42:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-14T13:42:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 4 2024-04-14T13:42:38.265 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:42:38.265 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:42:38.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 203359 11.57 MB/sec execute 312 sec latency 42.252 ms 2024-04-14T13:42:38.585 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:38 smithi008 ceph-mon[67070]: from='client.44147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:42:38.585 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:38 smithi008 ceph-mon[67070]: pgmap v60: 97 pgs: 97 active+clean; 346 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.9 MiB/s wr, 568 op/s 2024-04-14T13:42:38.586 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:38 smithi008 ceph-mon[67070]: from='client.44151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:42:38.586 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:38 smithi008 ceph-mon[67070]: osd.0 marked itself down and dead 2024-04-14T13:42:38.586 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:38 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/134118728' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:42:38.773 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:42:38.773 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:42:38.773 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:42:38.773 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:42:38.773 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-14T13:42:38.773 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-04-14T13:42:38.773 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-14T13:42:38.774 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-04-14T13:42:38.774 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:42:38.774 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "6/23 daemons upgraded", 2024-04-14T13:42:38.774 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T13:42:38.774 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:42:38.774 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:42:38.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:38 smithi012 ceph-mon[53289]: from='client.44147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:42:38.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:38 smithi012 ceph-mon[53289]: pgmap v60: 97 pgs: 97 active+clean; 346 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.9 MiB/s wr, 568 op/s 2024-04-14T13:42:38.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:38 smithi012 ceph-mon[53289]: from='client.44151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:42:38.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:38 smithi012 ceph-mon[53289]: osd.0 marked itself down and dead 2024-04-14T13:42:38.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:38 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/134118728' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:42:39.102 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:39 smithi008 podman[72600]: 2024-04-14 13:42:39.068213344 +0000 UTC m=+1.761632315 container died 3b8cdea7890cd57bac3da237c6377f4d130979e252ec041d183b8718083ea115 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , ceph=True, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, release=754, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, distribution-scope=public) 2024-04-14T13:42:39.275 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down 2024-04-14T13:42:39.275 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:42:39.275 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-14T13:42:39.275 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-14T13:42:39.275 INFO:teuthology.orchestra.run.smithi008.stdout: osd.0 (root=default,host=smithi008) is down 2024-04-14T13:42:39.350 INFO:teuthology.orchestra.run.smithi008.stdout: 5 203543 11.54 MB/sec execute 313 sec latency 934.231 ms 2024-04-14T13:42:39.526 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:39 smithi008 ceph-mon[67070]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T13:42:39.526 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:39 smithi008 ceph-mon[67070]: osdmap e74: 8 total, 7 up, 8 in 2024-04-14T13:42:39.527 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:39 smithi008 ceph-mon[67070]: from='client.44159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:42:39.527 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:39 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/2265568113' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:42:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:39 smithi012 ceph-mon[53289]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T13:42:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:39 smithi012 ceph-mon[53289]: osdmap e74: 8 total, 7 up, 8 in 2024-04-14T13:42:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:39 smithi012 ceph-mon[53289]: from='client.44159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:42:39.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:39 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/2265568113' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:42:40.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 203925 11.52 MB/sec execute 314 sec latency 1002.535 ms 2024-04-14T13:42:40.536 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:40 smithi008 ceph-mon[67070]: pgmap v62: 97 pgs: 36 peering, 61 active+clean; 346 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 6.2 MiB/s wr, 618 op/s 2024-04-14T13:42:40.536 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:40 smithi008 ceph-mon[67070]: Health check failed: Reduced data availability: 5 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-04-14T13:42:40.537 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:40 smithi008 ceph-mon[67070]: osdmap e75: 8 total, 7 up, 8 in 2024-04-14T13:42:40.537 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:40 smithi008 podman[72600]: 2024-04-14 13:42:40.298984502 +0000 UTC m=+2.992403472 container cleanup 3b8cdea7890cd57bac3da237c6377f4d130979e252ec041d183b8718083ea115 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, distribution-scope=public, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, name=centos-stream, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, vcs-type=git) 2024-04-14T13:42:40.537 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:40 smithi008 bash[72600]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0 2024-04-14T13:42:40.792 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:40 smithi008 podman[72685]: 2024-04-14 13:42:40.620782152 +0000 UTC m=+1.538931412 container remove 3b8cdea7890cd57bac3da237c6377f4d130979e252ec041d183b8718083ea115 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0, io.openshift.expose-services=, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, release=754, vcs-type=git, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, RELEASE=HEAD, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, version=8, GIT_BRANCH=HEAD, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-14T13:42:40.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:40 smithi012 ceph-mon[53289]: pgmap v62: 97 pgs: 36 peering, 61 active+clean; 346 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 6.2 MiB/s wr, 618 op/s 2024-04-14T13:42:40.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:40 smithi012 ceph-mon[53289]: Health check failed: Reduced data availability: 5 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-04-14T13:42:40.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:40 smithi012 ceph-mon[53289]: osdmap e75: 8 total, 7 up, 8 in 2024-04-14T13:42:41.296 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:41 smithi008 podman[72800]: 2024-04-14 13:42:41.135139232 +0000 UTC m=+0.023124217 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:42:41.296 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:41 smithi008 podman[72800]: 2024-04-14 13:42:41.26180776 +0000 UTC m=+0.149792742 container create f4c266bdfb49e0aaa51b62fc1b2efb573c83d0911099fed3912f959096dc8350 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T13:42:41.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 204297 11.53 MB/sec execute 315 sec latency 40.529 ms 2024-04-14T13:42:41.809 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:41 smithi008 podman[72800]: 2024-04-14 13:42:41.583104812 +0000 UTC m=+0.471089803 container init f4c266bdfb49e0aaa51b62fc1b2efb573c83d0911099fed3912f959096dc8350 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-14T13:42:41.809 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:41 smithi008 podman[72800]: 2024-04-14 13:42:41.589980412 +0000 UTC m=+0.477965394 container start f4c266bdfb49e0aaa51b62fc1b2efb573c83d0911099fed3912f959096dc8350 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403) 2024-04-14T13:42:41.809 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:41 smithi008 podman[72800]: 2024-04-14 13:42:41.784254905 +0000 UTC m=+0.672239894 container attach f4c266bdfb49e0aaa51b62fc1b2efb573c83d0911099fed3912f959096dc8350 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-14T13:42:42.320 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:42 smithi008 podman[72825]: 2024-04-14 13:42:42.084048323 +0000 UTC m=+0.030635613 container died f4c266bdfb49e0aaa51b62fc1b2efb573c83d0911099fed3912f959096dc8350 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-14T13:42:42.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 204811 11.53 MB/sec execute 316 sec latency 36.013 ms 2024-04-14T13:42:42.579 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:42 smithi008 ceph-mon[67070]: pgmap v64: 97 pgs: 36 peering, 61 active+clean; 346 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 7.8 MiB/s wr, 773 op/s 2024-04-14T13:42:42.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:42 smithi012 ceph-mon[53289]: pgmap v64: 97 pgs: 36 peering, 61 active+clean; 346 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 7.8 MiB/s wr, 773 op/s 2024-04-14T13:42:43.334 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:43 smithi008 podman[72825]: 2024-04-14 13:42:43.080095951 +0000 UTC m=+1.026683242 container remove f4c266bdfb49e0aaa51b62fc1b2efb573c83d0911099fed3912f959096dc8350 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-14T13:42:43.334 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:43 smithi008 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-14T13:42:43.334 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:43 smithi008 systemd[1]: Stopped Ceph osd.0 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:42:43.334 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:43 smithi008 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.0.service: Consumed 3min 42.734s CPU time. 2024-04-14T13:42:43.350 INFO:teuthology.orchestra.run.smithi008.stdout: 5 205262 11.52 MB/sec execute 317 sec latency 33.703 ms 2024-04-14T13:42:43.585 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:43 smithi008 systemd[1]: Starting Ceph osd.0 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:42:43.836 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:43 smithi008 podman[72923]: 2024-04-14 13:42:43.582705809 +0000 UTC m=+0.024653250 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:42:43.837 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:43 smithi008 podman[72923]: 2024-04-14 13:42:43.748315105 +0000 UTC m=+0.190262534 container create 275d947675617565eedcf89dad7b337520c16740dedc0bb8764dff0eb6a32dd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403) 2024-04-14T13:42:44.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 205724 11.55 MB/sec execute 318 sec latency 49.665 ms 2024-04-14T13:42:44.352 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:44 smithi008 podman[72923]: 2024-04-14 13:42:44.103145326 +0000 UTC m=+0.545092758 container init 275d947675617565eedcf89dad7b337520c16740dedc0bb8764dff0eb6a32dd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-activate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-14T13:42:44.352 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:44 smithi008 podman[72923]: 2024-04-14 13:42:44.110720289 +0000 UTC m=+0.552667722 container start 275d947675617565eedcf89dad7b337520c16740dedc0bb8764dff0eb6a32dd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-14T13:42:44.353 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:44 smithi008 podman[72923]: 2024-04-14 13:42:44.273411999 +0000 UTC m=+0.715359421 container attach 275d947675617565eedcf89dad7b337520c16740dedc0bb8764dff0eb6a32dd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-activate, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T13:42:44.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:44 smithi008 ceph-mon[67070]: pgmap v65: 97 pgs: 36 peering, 61 active+clean; 346 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 4.6 MiB/s wr, 451 op/s 2024-04-14T13:42:44.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:44 smithi012 ceph-mon[53289]: pgmap v65: 97 pgs: 36 peering, 61 active+clean; 346 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 4.6 MiB/s wr, 451 op/s 2024-04-14T13:42:44.856 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:44 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-activate[72937]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-14T13:42:44.856 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:44 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-activate[72937]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-14T13:42:44.856 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:44 smithi008 bash[72923]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-14T13:42:44.856 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:44 smithi008 bash[72923]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-14T13:42:44.856 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:44 smithi008 bash[72923]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-14T13:42:44.856 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:44 smithi008 bash[72923]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-14T13:42:44.856 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:44 smithi008 bash[72923]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-14T13:42:44.856 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:44 smithi008 bash[72923]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-14T13:42:44.856 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:44 smithi008 bash[72923]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-14T13:42:44.856 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:44 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-activate[72937]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-14T13:42:44.856 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:44 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-activate[72937]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-14T13:42:44.857 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:44 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-activate[72937]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-14T13:42:44.857 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:44 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-activate[72937]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-14T13:42:44.857 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:44 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-activate[72937]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-14T13:42:44.857 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:44 smithi008 podman[72923]: 2024-04-14 13:42:44.741402055 +0000 UTC m=+1.183349489 container died 275d947675617565eedcf89dad7b337520c16740dedc0bb8764dff0eb6a32dd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-activate, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240403, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-14T13:42:45.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 206254 11.57 MB/sec execute 319 sec latency 38.805 ms 2024-04-14T13:42:45.614 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:45 smithi008 ceph-mon[67070]: Health check failed: Degraded data redundancy: 236/1698 objects degraded (13.899%), 28 pgs degraded (PG_DEGRADED) 2024-04-14T13:42:45.615 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:45 smithi008 ceph-mon[67070]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 13 pgs peering) 2024-04-14T13:42:45.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:45 smithi012 ceph-mon[53289]: Health check failed: Degraded data redundancy: 236/1698 objects degraded (13.899%), 28 pgs degraded (PG_DEGRADED) 2024-04-14T13:42:45.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:45 smithi012 ceph-mon[53289]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 13 pgs peering) 2024-04-14T13:42:45.868 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:45 smithi008 podman[73073]: 2024-04-14 13:42:45.721308031 +0000 UTC m=+0.974814794 container remove 275d947675617565eedcf89dad7b337520c16740dedc0bb8764dff0eb6a32dd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-14T13:42:46.120 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:45 smithi008 podman[73128]: 2024-04-14 13:42:45.880386956 +0000 UTC m=+0.027327701 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:42:46.120 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:46 smithi008 podman[73128]: 2024-04-14 13:42:46.066968524 +0000 UTC m=+0.213909319 container create 29997ccc66acd144ae05a4a5790218e0c363d45c0e56d235ce154b72985a5743 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:42:46.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 206642 11.56 MB/sec execute 320 sec latency 35.196 ms 2024-04-14T13:42:46.630 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:46 smithi008 ceph-mon[67070]: pgmap v66: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 838 op/s; 236/1698 objects degraded (13.899%) 2024-04-14T13:42:46.631 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:46 smithi008 podman[73128]: 2024-04-14 13:42:46.387376387 +0000 UTC m=+0.534317142 container init 29997ccc66acd144ae05a4a5790218e0c363d45c0e56d235ce154b72985a5743 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403) 2024-04-14T13:42:46.631 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:46 smithi008 podman[73128]: 2024-04-14 13:42:46.398156011 +0000 UTC m=+0.545096764 container start 29997ccc66acd144ae05a4a5790218e0c363d45c0e56d235ce154b72985a5743 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0) 2024-04-14T13:42:46.631 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:46 smithi008 ceph-osd[73146]: -- 172.21.15.8:0/3301040129 <== mon.0 v2:172.21.15.8:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d432727680 con 0x55d4326f0c00 2024-04-14T13:42:46.631 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:46 smithi008 bash[73128]: 29997ccc66acd144ae05a4a5790218e0c363d45c0e56d235ce154b72985a5743 2024-04-14T13:42:46.631 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:46 smithi008 systemd[1]: Started Ceph osd.0 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:42:46.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:46 smithi012 ceph-mon[53289]: pgmap v66: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 838 op/s; 236/1698 objects degraded (13.899%) 2024-04-14T13:42:47.136 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:46 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0[73142]: 2024-04-14T13:42:46.980+0000 7f4b4097e7c0 -1 Falling back to public interface 2024-04-14T13:42:47.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 207013 11.55 MB/sec execute 321 sec latency 40.659 ms 2024-04-14T13:42:47.647 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:47 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:47.647 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:47 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:47.648 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:47 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:42:48.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:47 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:48.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:47 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:48.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:47 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:42:48.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 207252 11.52 MB/sec execute 322 sec latency 55.695 ms 2024-04-14T13:42:48.692 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:48 smithi008 ceph-mon[67070]: pgmap v67: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.8 MiB/s wr, 767 op/s; 236/1698 objects degraded (13.899%) 2024-04-14T13:42:48.692 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:48.692 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:42:49.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:48 smithi012 ceph-mon[53289]: pgmap v67: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.8 MiB/s wr, 767 op/s; 236/1698 objects degraded (13.899%) 2024-04-14T13:42:49.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:49.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:42:49.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 207478 11.50 MB/sec execute 323 sec latency 61.437 ms 2024-04-14T13:42:50.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 207713 11.49 MB/sec execute 324 sec latency 50.010 ms 2024-04-14T13:42:50.763 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:50 smithi008 ceph-mon[67070]: pgmap v68: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 344 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 9.8 KiB/s rd, 5.8 MiB/s wr, 583 op/s; 230/1674 objects degraded (13.740%) 2024-04-14T13:42:51.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:50 smithi012 ceph-mon[53289]: pgmap v68: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 344 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 9.8 KiB/s rd, 5.8 MiB/s wr, 583 op/s; 230/1674 objects degraded (13.740%) 2024-04-14T13:42:51.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 207930 11.48 MB/sec execute 325 sec latency 75.340 ms 2024-04-14T13:42:52.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 208212 11.46 MB/sec execute 326 sec latency 77.201 ms 2024-04-14T13:42:52.823 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:52 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0[73142]: 2024-04-14T13:42:52.697+0000 7f4b4097e7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-04-14T13:42:52.823 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:52 smithi008 ceph-mon[67070]: pgmap v69: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 344 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.0 MiB/s wr, 497 op/s; 230/1674 objects degraded (13.740%) 2024-04-14T13:42:52.823 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:52 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 230/1674 objects degraded (13.740%), 28 pgs degraded (PG_DEGRADED) 2024-04-14T13:42:53.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:52 smithi012 ceph-mon[53289]: pgmap v69: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 344 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.0 MiB/s wr, 497 op/s; 230/1674 objects degraded (13.740%) 2024-04-14T13:42:53.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:52 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 230/1674 objects degraded (13.740%), 28 pgs degraded (PG_DEGRADED) 2024-04-14T13:42:53.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 208521 11.44 MB/sec execute 327 sec latency 43.609 ms 2024-04-14T13:42:54.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 208755 11.43 MB/sec execute 328 sec latency 54.440 ms 2024-04-14T13:42:54.380 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:54 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0[73142]: 2024-04-14T13:42:54.261+0000 7f4b4097e7c0 -1 osd.0 73 log_to_monitors true 2024-04-14T13:42:54.893 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:42:54 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0[73142]: 2024-04-14T13:42:54.661+0000 7f4b31dd6700 -1 osd.0 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:42:54.894 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:54 smithi008 ceph-mon[67070]: pgmap v70: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 344 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.9 MiB/s wr, 486 op/s; 230/1674 objects degraded (13.740%) 2024-04-14T13:42:54.894 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:54 smithi008 ceph-mon[67070]: from='osd.0 [v2:172.21.15.8:6802/2115192120,v1:172.21.15.8:6803/2115192120]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T13:42:54.894 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:54 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:54.894 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:54 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:55.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:54 smithi012 ceph-mon[53289]: pgmap v70: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 344 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.9 MiB/s wr, 486 op/s; 230/1674 objects degraded (13.740%) 2024-04-14T13:42:55.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:54 smithi012 ceph-mon[53289]: from='osd.0 [v2:172.21.15.8:6802/2115192120,v1:172.21.15.8:6803/2115192120]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T13:42:55.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:54 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:55.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:54 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:55.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 208973 11.41 MB/sec execute 329 sec latency 65.951 ms 2024-04-14T13:42:56.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:55 smithi012 ceph-mon[53289]: from='osd.0 [v2:172.21.15.8:6802/2115192120,v1:172.21.15.8:6803/2115192120]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-14T13:42:56.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:55 smithi012 ceph-mon[53289]: osdmap e76: 8 total, 7 up, 8 in 2024-04-14T13:42:56.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:55 smithi012 ceph-mon[53289]: from='osd.0 [v2:172.21.15.8:6802/2115192120,v1:172.21.15.8:6803/2115192120]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:42:56.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:55 smithi008 ceph-mon[67070]: from='osd.0 [v2:172.21.15.8:6802/2115192120,v1:172.21.15.8:6803/2115192120]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-14T13:42:56.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:55 smithi008 ceph-mon[67070]: osdmap e76: 8 total, 7 up, 8 in 2024-04-14T13:42:56.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:55 smithi008 ceph-mon[67070]: from='osd.0 [v2:172.21.15.8:6802/2115192120,v1:172.21.15.8:6803/2115192120]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:42:56.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 209102 11.39 MB/sec execute 330 sec latency 661.456 ms 2024-04-14T13:42:56.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:56 smithi008 ceph-mon[67070]: pgmap v72: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.4 MiB/s wr, 457 op/s; 216/1587 objects degraded (13.611%) 2024-04-14T13:42:56.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:56 smithi008 ceph-mon[67070]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T13:42:56.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:56 smithi008 ceph-mon[67070]: osd.0 [v2:172.21.15.8:6802/2115192120,v1:172.21.15.8:6803/2115192120] boot 2024-04-14T13:42:56.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:56 smithi008 ceph-mon[67070]: osdmap e77: 8 total, 8 up, 8 in 2024-04-14T13:42:56.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:56 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:42:56.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:56 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:56.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:56 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:57.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:56 smithi012 ceph-mon[53289]: pgmap v72: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.4 MiB/s wr, 457 op/s; 216/1587 objects degraded (13.611%) 2024-04-14T13:42:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:56 smithi012 ceph-mon[53289]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T13:42:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:56 smithi012 ceph-mon[53289]: osd.0 [v2:172.21.15.8:6802/2115192120,v1:172.21.15.8:6803/2115192120] boot 2024-04-14T13:42:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:56 smithi012 ceph-mon[53289]: osdmap e77: 8 total, 8 up, 8 in 2024-04-14T13:42:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:56 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:42:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:56 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:56 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:42:57.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 209111 11.36 MB/sec execute 331 sec latency 1661.490 ms 2024-04-14T13:42:57.822 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:57 smithi008 ceph-mon[67070]: osdmap e78: 8 total, 8 up, 8 in 2024-04-14T13:42:57.822 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:57 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 216/1587 objects degraded (13.611%), 28 pgs degraded (PG_DEGRADED) 2024-04-14T13:42:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:57 smithi012 ceph-mon[53289]: osdmap e78: 8 total, 8 up, 8 in 2024-04-14T13:42:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:57 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 216/1587 objects degraded (13.611%), 28 pgs degraded (PG_DEGRADED) 2024-04-14T13:42:58.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 209118 11.33 MB/sec execute 332 sec latency 2607.861 ms 2024-04-14T13:42:58.894 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:58 smithi008 ceph-mon[67070]: pgmap v75: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.8 MiB/s wr, 306 op/s; 216/1587 objects degraded (13.611%) 2024-04-14T13:42:58.895 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:58 smithi008 ceph-mon[67070]: osdmap e79: 8 total, 8 up, 8 in 2024-04-14T13:42:59.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:58 smithi012 ceph-mon[53289]: pgmap v75: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.8 MiB/s wr, 306 op/s; 216/1587 objects degraded (13.611%) 2024-04-14T13:42:59.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:58 smithi012 ceph-mon[53289]: osdmap e79: 8 total, 8 up, 8 in 2024-04-14T13:42:59.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 209442 11.33 MB/sec execute 333 sec latency 672.873 ms 2024-04-14T13:42:59.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:42:59 smithi008 ceph-mon[67070]: osdmap e80: 8 total, 8 up, 8 in 2024-04-14T13:43:00.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:42:59 smithi012 ceph-mon[53289]: osdmap e80: 8 total, 8 up, 8 in 2024-04-14T13:43:00.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 209820 11.33 MB/sec execute 334 sec latency 1088.445 ms 2024-04-14T13:43:00.939 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:00 smithi008 ceph-mon[67070]: pgmap v78: 97 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 2 peering, 4 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+undersized, 79 active+clean; 341 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.9 MiB/s wr, 252 op/s; 197/1557 objects degraded (12.653%); 1.0 MiB/s, 20 objects/s recovering 2024-04-14T13:43:00.939 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:00 smithi008 ceph-mon[67070]: osdmap e81: 8 total, 8 up, 8 in 2024-04-14T13:43:00.939 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:00 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:00.939 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:00 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:00.939 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:00 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:43:00.939 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:00 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:43:00.939 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:00 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:00.939 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:00 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:00.940 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:00 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:43:00.940 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:00 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:43:00.940 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:00 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:43:00.940 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:00 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:43:00.940 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:00 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T13:43:01.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:00 smithi012 ceph-mon[53289]: pgmap v78: 97 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 2 peering, 4 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+undersized, 79 active+clean; 341 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.9 MiB/s wr, 252 op/s; 197/1557 objects degraded (12.653%); 1.0 MiB/s, 20 objects/s recovering 2024-04-14T13:43:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:00 smithi012 ceph-mon[53289]: osdmap e81: 8 total, 8 up, 8 in 2024-04-14T13:43:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:00 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:00 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:00 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:43:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:00 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:43:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:00 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:00 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:00 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:43:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:00 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:43:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:00 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:43:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:00 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:43:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:00 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T13:43:01.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 210109 11.32 MB/sec execute 335 sec latency 651.661 ms 2024-04-14T13:43:01.695 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:01 smithi008 ceph-mon[67070]: pgmap v80: 97 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 2 peering, 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.3 MiB/s wr, 272 op/s; 163/1554 objects degraded (10.489%); 1.1 MiB/s, 31 objects/s recovering 2024-04-14T13:43:01.696 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:01 smithi008 ceph-mon[67070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T13:43:01.696 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:01 smithi008 ceph-mon[67070]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-04-14T13:43:01.696 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:01 smithi008 ceph-mon[67070]: osdmap e82: 8 total, 8 up, 8 in 2024-04-14T13:43:02.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:01 smithi012 ceph-mon[53289]: pgmap v80: 97 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 2 peering, 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.3 MiB/s wr, 272 op/s; 163/1554 objects degraded (10.489%); 1.1 MiB/s, 31 objects/s recovering 2024-04-14T13:43:02.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:01 smithi012 ceph-mon[53289]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T13:43:02.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:01 smithi012 ceph-mon[53289]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-04-14T13:43:02.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:01 smithi012 ceph-mon[53289]: osdmap e82: 8 total, 8 up, 8 in 2024-04-14T13:43:02.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 210368 11.30 MB/sec execute 336 sec latency 647.082 ms 2024-04-14T13:43:02.708 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:02 smithi008 ceph-mon[67070]: osdmap e83: 8 total, 8 up, 8 in 2024-04-14T13:43:02.708 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:02 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 163/1554 objects degraded (10.489%), 7 pgs degraded (PG_DEGRADED) 2024-04-14T13:43:03.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:02 smithi012 ceph-mon[53289]: osdmap e83: 8 total, 8 up, 8 in 2024-04-14T13:43:03.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:02 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 163/1554 objects degraded (10.489%), 7 pgs degraded (PG_DEGRADED) 2024-04-14T13:43:03.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 210611 11.27 MB/sec execute 337 sec latency 1027.848 ms 2024-04-14T13:43:03.742 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:03 smithi008 ceph-mon[67070]: pgmap v83: 97 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 2 peering, 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.3 MiB/s wr, 273 op/s; 163/1554 objects degraded (10.489%); 1.1 MiB/s, 31 objects/s recovering 2024-04-14T13:43:03.742 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:03 smithi008 ceph-mon[67070]: osdmap e84: 8 total, 8 up, 8 in 2024-04-14T13:43:03.742 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:03.742 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:43:04.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:03 smithi012 ceph-mon[53289]: pgmap v83: 97 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 2 peering, 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.3 MiB/s wr, 273 op/s; 163/1554 objects degraded (10.489%); 1.1 MiB/s, 31 objects/s recovering 2024-04-14T13:43:04.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:03 smithi012 ceph-mon[53289]: osdmap e84: 8 total, 8 up, 8 in 2024-04-14T13:43:04.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:04.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:43:04.351 INFO:teuthology.orchestra.run.smithi008.stdout: 5 210829 11.26 MB/sec execute 338 sec latency 362.807 ms 2024-04-14T13:43:04.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:04 smithi008 ceph-mon[67070]: osdmap e85: 8 total, 8 up, 8 in 2024-04-14T13:43:04.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:04 smithi008 ceph-mon[67070]: osdmap e86: 8 total, 8 up, 8 in 2024-04-14T13:43:05.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:04 smithi012 ceph-mon[53289]: osdmap e85: 8 total, 8 up, 8 in 2024-04-14T13:43:05.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:04 smithi012 ceph-mon[53289]: osdmap e86: 8 total, 8 up, 8 in 2024-04-14T13:43:05.352 INFO:teuthology.orchestra.run.smithi008.stdout: 5 211080 11.24 MB/sec execute 339 sec latency 737.524 ms 2024-04-14T13:43:05.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:05 smithi008 ceph-mon[67070]: pgmap v86: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 93 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.3 MiB/s wr, 485 op/s; 88/1653 objects degraded (5.324%); 0 B/s, 18 objects/s recovering 2024-04-14T13:43:06.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:05 smithi012 ceph-mon[53289]: pgmap v86: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 93 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.3 MiB/s wr, 485 op/s; 88/1653 objects degraded (5.324%); 0 B/s, 18 objects/s recovering 2024-04-14T13:43:06.352 INFO:teuthology.orchestra.run.smithi008.stdout: 5 211390 11.25 MB/sec execute 340 sec latency 108.996 ms 2024-04-14T13:43:06.835 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:06 smithi008 ceph-mon[67070]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 88/1653 objects degraded (5.324%), 4 pgs degraded) 2024-04-14T13:43:07.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:06 smithi012 ceph-mon[53289]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 88/1653 objects degraded (5.324%), 4 pgs degraded) 2024-04-14T13:43:07.352 INFO:teuthology.orchestra.run.smithi008.stdout: 5 211720 11.23 MB/sec execute 341 sec latency 51.098 ms 2024-04-14T13:43:07.859 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:07 smithi008 ceph-mon[67070]: pgmap v88: 97 pgs: 1 peering, 96 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 443 op/s; 0 B/s, 34 objects/s recovering 2024-04-14T13:43:08.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:07 smithi012 ceph-mon[53289]: pgmap v88: 97 pgs: 1 peering, 96 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 443 op/s; 0 B/s, 34 objects/s recovering 2024-04-14T13:43:08.352 INFO:teuthology.orchestra.run.smithi008.stdout: 5 212058 11.23 MB/sec execute 342 sec latency 49.852 ms 2024-04-14T13:43:09.352 INFO:teuthology.orchestra.run.smithi008.stdout: 5 212298 11.21 MB/sec execute 343 sec latency 57.522 ms 2024-04-14T13:43:09.775 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:43:09.886 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:09 smithi008 ceph-mon[67070]: pgmap v89: 97 pgs: 1 peering, 96 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.8 KiB/s rd, 3.4 MiB/s wr, 350 op/s; 0 B/s, 27 objects/s recovering 2024-04-14T13:43:10.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:09 smithi012 ceph-mon[53289]: pgmap v89: 97 pgs: 1 peering, 96 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.8 KiB/s rd, 3.4 MiB/s wr, 350 op/s; 0 B/s, 27 objects/s recovering 2024-04-14T13:43:10.352 INFO:teuthology.orchestra.run.smithi008.stdout: 5 212628 11.21 MB/sec execute 344 sec latency 50.279 ms 2024-04-14T13:43:10.707 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:43:10.707 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (5m) 16s ago 14m 15.2M - 0.23.0 ba2b418f427c cfddb9a7f858 2024-04-14T13:43:10.708 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (64s) 16s ago 14m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 1e69eabdcc36 2024-04-14T13:43:10.708 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (57s) 52s ago 13m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 60c7a2992eb0 2024-04-14T13:43:10.708 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 16s ago 13m - - 2024-04-14T13:43:10.708 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (8m) 16s ago 8m 38.4M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:43:10.708 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (8m) 52s ago 8m 14.4M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:43:10.708 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (3m) 16s ago 15m 503M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:43:10.708 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (2m) 52s ago 13m 451M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:43:10.708 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (2m) 16s ago 15m 45.6M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 673dd6f4c8d8 2024-04-14T13:43:10.708 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (2m) 52s ago 13m 35.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 d498dec29c4f 2024-04-14T13:43:10.708 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (7m) 16s ago 7m 281M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:43:10.708 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (7m) 52s ago 7m 88.5M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:43:10.708 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (4m) 16s ago 13m 14.0M - 1.3.1 1dbe0e931976 646dfca70be4 2024-04-14T13:43:10.708 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (4m) 52s ago 13m 18.4M - 1.3.1 1dbe0e931976 0a60dcd4044c 2024-04-14T13:43:10.708 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (24s) 16s ago 12m 11.8M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 29997ccc66ac 2024-04-14T13:43:10.708 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (12m) 16s ago 12m 354M 1937M 17.2.0 e1d6a67b021e be86642c7790 2024-04-14T13:43:10.709 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (11m) 16s ago 11m 487M 1937M 17.2.0 e1d6a67b021e b1017f00578f 2024-04-14T13:43:10.709 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (11m) 16s ago 11m 367M 1937M 17.2.0 e1d6a67b021e 66fca4b191be 2024-04-14T13:43:10.709 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (10m) 52s ago 10m 614M 2705M 17.2.0 e1d6a67b021e 43b02cb64951 2024-04-14T13:43:10.709 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (10m) 52s ago 10m 430M 2705M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:43:10.709 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (10m) 52s ago 10m 341M 2705M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:43:10.709 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (9m) 52s ago 9m 552M 2705M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:43:10.709 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (3m) 16s ago 12m 65.0M - 2.33.4 514e6a882f6e 12f97f75d620 2024-04-14T13:43:10.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:10 smithi008 ceph-mon[67070]: from='client.34226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:43:11.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:10 smithi012 ceph-mon[53289]: from='client.34226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:43:11.296 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:43:11.296 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:43:11.296 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:43:11.296 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:43:11.296 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:43:11.296 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:43:11.296 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:43:11.296 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:43:11.296 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-14T13:43:11.296 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T13:43:11.297 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:43:11.297 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:43:11.297 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:43:11.297 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:43:11.297 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:43:11.297 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-04-14T13:43:11.297 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 5 2024-04-14T13:43:11.297 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:43:11.297 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:43:11.352 INFO:teuthology.orchestra.run.smithi008.stdout: 5 212969 11.20 MB/sec execute 345 sec latency 33.601 ms 2024-04-14T13:43:11.776 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:11 smithi008 ceph-mon[67070]: from='client.34230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:43:11.776 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:11 smithi008 ceph-mon[67070]: pgmap v90: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.4 MiB/s wr, 557 op/s; 0 B/s, 21 objects/s recovering 2024-04-14T13:43:11.776 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:11 smithi008 ceph-mon[67070]: from='client.34234 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:43:11.776 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:11 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/3843799099' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:43:11.790 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:43:11.790 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:43:11.790 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:43:11.790 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:43:11.790 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-14T13:43:11.790 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-04-14T13:43:11.791 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-14T13:43:11.791 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-04-14T13:43:11.791 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:43:11.791 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "7/23 daemons upgraded", 2024-04-14T13:43:11.791 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T13:43:11.791 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:43:11.791 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:43:12.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:11 smithi012 ceph-mon[53289]: from='client.34230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:43:12.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:11 smithi012 ceph-mon[53289]: pgmap v90: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.4 MiB/s wr, 557 op/s; 0 B/s, 21 objects/s recovering 2024-04-14T13:43:12.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:11 smithi012 ceph-mon[53289]: from='client.34234 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:43:12.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:11 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/3843799099' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:43:12.352 INFO:teuthology.orchestra.run.smithi008.stdout: 5 213328 11.21 MB/sec execute 346 sec latency 52.815 ms 2024-04-14T13:43:12.354 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T13:43:12.354 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:43:12.354 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-14T13:43:13.014 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:12 smithi008 ceph-mon[67070]: from='client.34242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:43:13.015 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:12 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/3719072724' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:43:13.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:12 smithi012 ceph-mon[53289]: from='client.34242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:43:13.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:12 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/3719072724' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:43:13.352 INFO:teuthology.orchestra.run.smithi008.stdout: 5 213686 11.20 MB/sec execute 347 sec latency 47.574 ms 2024-04-14T13:43:14.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:13 smithi008 ceph-mon[67070]: pgmap v91: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.7 MiB/s wr, 493 op/s; 0 B/s, 18 objects/s recovering 2024-04-14T13:43:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:13 smithi012 ceph-mon[53289]: pgmap v91: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.7 MiB/s wr, 493 op/s; 0 B/s, 18 objects/s recovering 2024-04-14T13:43:14.352 INFO:teuthology.orchestra.run.smithi008.stdout: 5 214030 11.19 MB/sec execute 348 sec latency 52.888 ms 2024-04-14T13:43:15.352 INFO:teuthology.orchestra.run.smithi008.stdout: 5 214391 11.18 MB/sec execute 349 sec latency 37.799 ms 2024-04-14T13:43:15.803 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:15 smithi008 ceph-mon[67070]: pgmap v92: 97 pgs: 97 active+clean; 343 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.6 MiB/s wr, 460 op/s; 0 B/s, 8 objects/s recovering 2024-04-14T13:43:15.803 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:15 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T13:43:16.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:15 smithi012 ceph-mon[53289]: pgmap v92: 97 pgs: 97 active+clean; 343 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.6 MiB/s wr, 460 op/s; 0 B/s, 8 objects/s recovering 2024-04-14T13:43:16.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:15 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T13:43:16.352 INFO:teuthology.orchestra.run.smithi008.stdout: 5 214761 11.18 MB/sec execute 350 sec latency 38.932 ms 2024-04-14T13:43:16.842 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:16 smithi008 ceph-mon[67070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T13:43:16.843 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:16 smithi008 ceph-mon[67070]: Upgrade: osd.1 is safe to restart 2024-04-14T13:43:17.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:16 smithi012 ceph-mon[53289]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T13:43:17.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:16 smithi012 ceph-mon[53289]: Upgrade: osd.1 is safe to restart 2024-04-14T13:43:17.352 INFO:teuthology.orchestra.run.smithi008.stdout: 5 215117 11.18 MB/sec execute 351 sec latency 49.352 ms 2024-04-14T13:43:17.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:17 smithi008 ceph-mon[67070]: pgmap v93: 97 pgs: 97 active+clean; 342 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.0 MiB/s wr, 394 op/s 2024-04-14T13:43:18.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:17 smithi012 ceph-mon[53289]: pgmap v93: 97 pgs: 97 active+clean; 342 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.0 MiB/s wr, 394 op/s 2024-04-14T13:43:18.352 INFO:teuthology.orchestra.run.smithi008.stdout: 5 215606 11.18 MB/sec execute 352 sec latency 52.182 ms 2024-04-14T13:43:19.133 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:19 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:19.133 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:19 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:43:19.133 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:19 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:19.133 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:19 smithi008 ceph-mon[67070]: pgmap v94: 97 pgs: 97 active+clean; 342 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 385 op/s 2024-04-14T13:43:19.133 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:19 smithi008 ceph-mon[67070]: Upgrade: Updating osd.1 2024-04-14T13:43:19.133 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:19 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:19.133 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:19 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T13:43:19.133 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:19 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:43:19.133 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:19 smithi008 ceph-mon[67070]: Deploying daemon osd.1 on smithi008 2024-04-14T13:43:19.352 INFO:teuthology.orchestra.run.smithi008.stdout: 5 215886 11.16 MB/sec execute 353 sec latency 49.019 ms 2024-04-14T13:43:19.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:19 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:19.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:19 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:43:19.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:19 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:19.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:19 smithi012 ceph-mon[53289]: pgmap v94: 97 pgs: 97 active+clean; 342 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 385 op/s 2024-04-14T13:43:19.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:19 smithi012 ceph-mon[53289]: Upgrade: Updating osd.1 2024-04-14T13:43:19.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:19 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:19.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:19 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T13:43:19.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:19 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:43:19.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:19 smithi012 ceph-mon[53289]: Deploying daemon osd.1 on smithi008 2024-04-14T13:43:20.352 INFO:teuthology.orchestra.run.smithi008.stdout: 5 216317 11.17 MB/sec execute 354 sec latency 35.540 ms 2024-04-14T13:43:21.352 INFO:teuthology.orchestra.run.smithi008.stdout: 5 216700 11.17 MB/sec execute 355 sec latency 53.171 ms 2024-04-14T13:43:21.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:21 smithi008 ceph-mon[67070]: pgmap v95: 97 pgs: 97 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.1 MiB/s wr, 623 op/s 2024-04-14T13:43:21.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:21 smithi012 ceph-mon[53289]: pgmap v95: 97 pgs: 97 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.1 MiB/s wr, 623 op/s 2024-04-14T13:43:22.352 INFO:teuthology.orchestra.run.smithi008.stdout: 5 217131 11.17 MB/sec execute 356 sec latency 51.535 ms 2024-04-14T13:43:23.352 INFO:teuthology.orchestra.run.smithi008.stdout: 5 217655 11.18 MB/sec execute 357 sec latency 37.450 ms 2024-04-14T13:43:23.685 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:23 smithi008 ceph-mon[67070]: pgmap v96: 97 pgs: 97 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 439 op/s 2024-04-14T13:43:23.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:23 smithi012 ceph-mon[53289]: pgmap v96: 97 pgs: 97 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 439 op/s 2024-04-14T13:43:24.353 INFO:teuthology.orchestra.run.smithi008.stdout: 5 218185 11.18 MB/sec execute 358 sec latency 24.203 ms 2024-04-14T13:43:24.691 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:24 smithi008 systemd[1]: Stopping Ceph osd.1 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:43:25.205 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:24 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1[38101]: 2024-04-14T13:43:24.989+0000 7f7a4858d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:43:25.206 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:24 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1[38101]: 2024-04-14T13:43:24.989+0000 7f7a4858d700 -1 osd.1 86 *** Got signal Terminated *** 2024-04-14T13:43:25.206 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:24 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1[38101]: 2024-04-14T13:43:24.989+0000 7f7a4858d700 -1 osd.1 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T13:43:25.353 INFO:teuthology.orchestra.run.smithi008.stdout: 5 218633 11.20 MB/sec execute 359 sec latency 53.576 ms 2024-04-14T13:43:25.713 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:25 smithi008 ceph-mon[67070]: pgmap v97: 97 pgs: 97 active+clean; 352 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.8 MiB/s wr, 688 op/s 2024-04-14T13:43:25.714 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:25 smithi008 ceph-mon[67070]: osd.1 marked itself down and dead 2024-04-14T13:43:25.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:25 smithi012 ceph-mon[53289]: pgmap v97: 97 pgs: 97 active+clean; 352 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.8 MiB/s wr, 688 op/s 2024-04-14T13:43:25.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:25 smithi012 ceph-mon[53289]: osd.1 marked itself down and dead 2024-04-14T13:43:26.227 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:26 smithi008 podman[75238]: 2024-04-14 13:43:26.161055913 +0000 UTC m=+1.435926327 container died be86642c7790035744e9a5322b910cbf0eb6f1baec3ec87eb0e3763da305c28e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1, io.openshift.expose-services=, io.buildah.version=1.19.8, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, release=754, RELEASE=HEAD, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-14T13:43:26.353 INFO:teuthology.orchestra.run.smithi008.stdout: 5 218821 11.17 MB/sec execute 360 sec latency 798.841 ms 2024-04-14T13:43:26.751 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:26 smithi008 ceph-mon[67070]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T13:43:26.752 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:26 smithi008 ceph-mon[67070]: osdmap e87: 8 total, 7 up, 8 in 2024-04-14T13:43:26.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:26 smithi012 ceph-mon[53289]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T13:43:26.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:26 smithi012 ceph-mon[53289]: osdmap e87: 8 total, 7 up, 8 in 2024-04-14T13:43:27.273 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:27 smithi008 podman[75238]: 2024-04-14 13:43:27.187617761 +0000 UTC m=+2.462488179 container cleanup be86642c7790035744e9a5322b910cbf0eb6f1baec3ec87eb0e3763da305c28e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1, GIT_CLEAN=True, RELEASE=HEAD, vendor=Red Hat, Inc., version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.openshift.tags=base centos centos-stream, name=centos-stream, maintainer=Guillaume Abrioux ) 2024-04-14T13:43:27.274 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:27 smithi008 bash[75238]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1 2024-04-14T13:43:27.353 INFO:teuthology.orchestra.run.smithi008.stdout: 5 219105 11.16 MB/sec execute 361 sec latency 999.366 ms 2024-04-14T13:43:27.529 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:27 smithi008 ceph-mon[67070]: pgmap v99: 97 pgs: 14 stale+active+clean, 83 active+clean; 352 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.2 MiB/s wr, 610 op/s 2024-04-14T13:43:27.529 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:27 smithi008 ceph-mon[67070]: osdmap e88: 8 total, 7 up, 8 in 2024-04-14T13:43:27.782 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:27 smithi008 podman[75255]: 2024-04-14 13:43:27.59124837 +0000 UTC m=+1.417828160 container remove be86642c7790035744e9a5322b910cbf0eb6f1baec3ec87eb0e3763da305c28e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, release=754, io.openshift.expose-services=, RELEASE=HEAD, ceph=True, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, vcs-type=git, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, com.redhat.component=centos-stream-container) 2024-04-14T13:43:27.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:27 smithi012 ceph-mon[53289]: pgmap v99: 97 pgs: 14 stale+active+clean, 83 active+clean; 352 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.2 MiB/s wr, 610 op/s 2024-04-14T13:43:27.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:27 smithi012 ceph-mon[53289]: osdmap e88: 8 total, 7 up, 8 in 2024-04-14T13:43:28.304 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:28 smithi008 podman[75338]: 2024-04-14 13:43:28.080955886 +0000 UTC m=+0.027179253 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:43:28.304 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:28 smithi008 podman[75338]: 2024-04-14 13:43:28.192566584 +0000 UTC m=+0.138789930 container create 66c1a5fb1e125546d022d29f0647e51df8dde7f047e43b98b3d638d1b7cc04de (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-14T13:43:28.353 INFO:teuthology.orchestra.run.smithi008.stdout: 5 219385 11.15 MB/sec execute 362 sec latency 61.228 ms 2024-04-14T13:43:28.556 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:28 smithi008 podman[75338]: 2024-04-14 13:43:28.462207405 +0000 UTC m=+0.408430751 container init 66c1a5fb1e125546d022d29f0647e51df8dde7f047e43b98b3d638d1b7cc04de (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-14T13:43:28.556 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:28 smithi008 podman[75338]: 2024-04-14 13:43:28.468295114 +0000 UTC m=+0.414518469 container start 66c1a5fb1e125546d022d29f0647e51df8dde7f047e43b98b3d638d1b7cc04de (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux ) 2024-04-14T13:43:28.808 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:28 smithi008 podman[75338]: 2024-04-14 13:43:28.571428522 +0000 UTC m=+0.517651882 container attach 66c1a5fb1e125546d022d29f0647e51df8dde7f047e43b98b3d638d1b7cc04de (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, ceph=True) 2024-04-14T13:43:29.065 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:28 smithi008 podman[75363]: 2024-04-14 13:43:28.935207255 +0000 UTC m=+0.030613364 container died 66c1a5fb1e125546d022d29f0647e51df8dde7f047e43b98b3d638d1b7cc04de (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T13:43:29.353 INFO:teuthology.orchestra.run.smithi008.stdout: 5 219786 11.15 MB/sec execute 363 sec latency 43.583 ms 2024-04-14T13:43:29.571 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:29 smithi008 ceph-mon[67070]: pgmap v101: 97 pgs: 14 stale+active+clean, 83 active+clean; 352 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.4 MiB/s wr, 406 op/s 2024-04-14T13:43:29.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:29 smithi012 ceph-mon[53289]: pgmap v101: 97 pgs: 14 stale+active+clean, 83 active+clean; 352 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.4 MiB/s wr, 406 op/s 2024-04-14T13:43:30.324 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:30 smithi008 podman[75363]: 2024-04-14 13:43:30.164293871 +0000 UTC m=+1.259699976 container remove 66c1a5fb1e125546d022d29f0647e51df8dde7f047e43b98b3d638d1b7cc04de (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-14T13:43:30.324 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:30 smithi008 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.1.service: Deactivated successfully. 2024-04-14T13:43:30.324 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:30 smithi008 systemd[1]: Stopped Ceph osd.1 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:43:30.324 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:30 smithi008 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.1.service: Consumed 3min 58.152s CPU time. 2024-04-14T13:43:30.353 INFO:teuthology.orchestra.run.smithi008.stdout: 5 220256 11.14 MB/sec execute 364 sec latency 56.736 ms 2024-04-14T13:43:30.582 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:30 smithi008 ceph-mon[67070]: Health check failed: Degraded data redundancy: 173/1608 objects degraded (10.759%), 23 pgs degraded (PG_DEGRADED) 2024-04-14T13:43:30.582 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:30 smithi008 systemd[1]: Starting Ceph osd.1 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:43:30.836 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:30 smithi008 podman[75459]: 2024-04-14 13:43:30.693359179 +0000 UTC m=+0.025257286 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:43:30.837 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:30 smithi008 podman[75459]: 2024-04-14 13:43:30.82419359 +0000 UTC m=+0.156091698 container create 4db95ee6f4dbc31ccc76ba05a2e2adf5f9305723e9aff4bb0eff417c989e91f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-activate, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403) 2024-04-14T13:43:31.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:30 smithi012 ceph-mon[53289]: Health check failed: Degraded data redundancy: 173/1608 objects degraded (10.759%), 23 pgs degraded (PG_DEGRADED) 2024-04-14T13:43:31.340 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:31 smithi008 podman[75459]: 2024-04-14 13:43:31.228724649 +0000 UTC m=+0.560622754 container init 4db95ee6f4dbc31ccc76ba05a2e2adf5f9305723e9aff4bb0eff417c989e91f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-activate, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-14T13:43:31.340 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:31 smithi008 podman[75459]: 2024-04-14 13:43:31.235843865 +0000 UTC m=+0.567741971 container start 4db95ee6f4dbc31ccc76ba05a2e2adf5f9305723e9aff4bb0eff417c989e91f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-activate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:43:31.353 INFO:teuthology.orchestra.run.smithi008.stdout: 5 220680 11.16 MB/sec execute 365 sec latency 38.714 ms 2024-04-14T13:43:31.591 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:31 smithi008 ceph-mon[67070]: pgmap v102: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.1 MiB/s wr, 715 op/s; 173/1608 objects degraded (10.759%) 2024-04-14T13:43:31.591 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:31 smithi008 podman[75459]: 2024-04-14 13:43:31.447389386 +0000 UTC m=+0.779287493 container attach 4db95ee6f4dbc31ccc76ba05a2e2adf5f9305723e9aff4bb0eff417c989e91f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-activate, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:43:32.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:31 smithi012 ceph-mon[53289]: pgmap v102: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.1 MiB/s wr, 715 op/s; 173/1608 objects degraded (10.759%) 2024-04-14T13:43:32.103 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:31 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-activate[75473]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-14T13:43:32.103 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:31 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-activate[75473]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-14T13:43:32.103 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:31 smithi008 bash[75459]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-14T13:43:32.103 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:31 smithi008 bash[75459]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-14T13:43:32.103 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:31 smithi008 bash[75459]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-14T13:43:32.103 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:31 smithi008 bash[75459]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-14T13:43:32.104 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:31 smithi008 bash[75459]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-14T13:43:32.104 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:31 smithi008 bash[75459]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-14T13:43:32.104 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:31 smithi008 bash[75459]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-14T13:43:32.104 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:31 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-activate[75473]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-14T13:43:32.104 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:31 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-activate[75473]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-14T13:43:32.104 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:31 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-activate[75473]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-14T13:43:32.104 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:31 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-activate[75473]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-14T13:43:32.104 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:31 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-activate[75473]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-14T13:43:32.104 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:31 smithi008 podman[75459]: 2024-04-14 13:43:31.911209257 +0000 UTC m=+1.243107355 container died 4db95ee6f4dbc31ccc76ba05a2e2adf5f9305723e9aff4bb0eff417c989e91f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-14T13:43:32.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 221119 11.17 MB/sec execute 366 sec latency 40.224 ms 2024-04-14T13:43:33.114 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:33 smithi008 podman[75604]: 2024-04-14 13:43:33.089657775 +0000 UTC m=+1.171763520 container remove 4db95ee6f4dbc31ccc76ba05a2e2adf5f9305723e9aff4bb0eff417c989e91f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-04-14T13:43:33.353 INFO:teuthology.orchestra.run.smithi008.stdout: 5 221530 11.16 MB/sec execute 367 sec latency 39.263 ms 2024-04-14T13:43:33.365 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:33 smithi008 ceph-mon[67070]: pgmap v103: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.4 MiB/s wr, 342 op/s; 173/1608 objects degraded (10.759%) 2024-04-14T13:43:33.365 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:33 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:33.365 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:33 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:43:33.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:33 smithi012 ceph-mon[53289]: pgmap v103: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.4 MiB/s wr, 342 op/s; 173/1608 objects degraded (10.759%) 2024-04-14T13:43:33.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:33 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:33.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:33 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:43:33.617 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:33 smithi008 podman[75659]: 2024-04-14 13:43:33.267478479 +0000 UTC m=+0.024302719 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:43:33.617 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:33 smithi008 podman[75659]: 2024-04-14 13:43:33.424694881 +0000 UTC m=+0.181519122 container create 97df88f10bd6aa144a1bda874a67d40ea31cf24c89f592ec5346f23e1fb72673 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-14T13:43:33.868 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:33 smithi008 podman[75659]: 2024-04-14 13:43:33.779299942 +0000 UTC m=+0.536124172 container init 97df88f10bd6aa144a1bda874a67d40ea31cf24c89f592ec5346f23e1fb72673 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:43:33.868 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:33 smithi008 podman[75659]: 2024-04-14 13:43:33.787204551 +0000 UTC m=+0.544028790 container start 97df88f10bd6aa144a1bda874a67d40ea31cf24c89f592ec5346f23e1fb72673 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403) 2024-04-14T13:43:33.868 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:33 smithi008 ceph-osd[75677]: -- 172.21.15.8:0/3997186579 <== mon.1 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x55f3bc157d40 con 0x55f3bc129000 2024-04-14T13:43:33.868 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:33 smithi008 ceph-osd[75677]: -- 172.21.15.8:0/3997186579 <== mon.1 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f3bcf02000 con 0x55f3bc129000 2024-04-14T13:43:34.119 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:33 smithi008 bash[75659]: 97df88f10bd6aa144a1bda874a67d40ea31cf24c89f592ec5346f23e1fb72673 2024-04-14T13:43:34.119 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:33 smithi008 systemd[1]: Started Ceph osd.1 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:43:34.353 INFO:teuthology.orchestra.run.smithi008.stdout: 5 221742 11.14 MB/sec execute 368 sec latency 67.984 ms 2024-04-14T13:43:34.370 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:34 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1[75673]: 2024-04-14T13:43:34.364+0000 7f3b8fb6d7c0 -1 Falling back to public interface 2024-04-14T13:43:35.134 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:34 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:35.134 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:34 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:35.134 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:34 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:43:35.134 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:34 smithi008 ceph-mon[67070]: pgmap v104: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 347 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.4 MiB/s wr, 579 op/s; 195/1722 objects degraded (11.324%) 2024-04-14T13:43:35.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:34 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:35.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:34 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:35.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:34 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:43:35.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:34 smithi012 ceph-mon[53289]: pgmap v104: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 347 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.4 MiB/s wr, 579 op/s; 195/1722 objects degraded (11.324%) 2024-04-14T13:43:35.353 INFO:teuthology.orchestra.run.smithi008.stdout: 5 221967 11.14 MB/sec execute 369 sec latency 61.257 ms 2024-04-14T13:43:36.353 INFO:teuthology.orchestra.run.smithi008.stdout: 5 222196 11.12 MB/sec execute 370 sec latency 96.306 ms 2024-04-14T13:43:36.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:36 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 197/1752 objects degraded (11.244%), 23 pgs degraded (PG_DEGRADED) 2024-04-14T13:43:37.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:36 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 197/1752 objects degraded (11.244%), 23 pgs degraded (PG_DEGRADED) 2024-04-14T13:43:37.353 INFO:teuthology.orchestra.run.smithi008.stdout: 5 222440 11.10 MB/sec execute 371 sec latency 103.016 ms 2024-04-14T13:43:37.702 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:37 smithi008 ceph-mon[67070]: pgmap v105: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 349 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.4 MiB/s wr, 538 op/s; 197/1752 objects degraded (11.244%) 2024-04-14T13:43:38.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:37 smithi012 ceph-mon[53289]: pgmap v105: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 349 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.4 MiB/s wr, 538 op/s; 197/1752 objects degraded (11.244%) 2024-04-14T13:43:38.353 INFO:teuthology.orchestra.run.smithi008.stdout: 5 222727 11.09 MB/sec execute 372 sec latency 48.019 ms 2024-04-14T13:43:39.353 INFO:teuthology.orchestra.run.smithi008.stdout: 5 223212 11.10 MB/sec execute 373 sec latency 67.020 ms 2024-04-14T13:43:39.741 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:39 smithi008 ceph-mon[67070]: pgmap v106: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 349 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.5 MiB/s wr, 450 op/s; 197/1752 objects degraded (11.244%) 2024-04-14T13:43:39.991 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:39 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1[75673]: 2024-04-14T13:43:39.909+0000 7f3b8fb6d7c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-04-14T13:43:40.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:39 smithi012 ceph-mon[53289]: pgmap v106: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 349 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.5 MiB/s wr, 450 op/s; 197/1752 objects degraded (11.244%) 2024-04-14T13:43:40.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 223790 11.10 MB/sec execute 374 sec latency 22.680 ms 2024-04-14T13:43:41.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 224314 11.12 MB/sec execute 375 sec latency 39.423 ms 2024-04-14T13:43:41.502 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:41 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1[75673]: 2024-04-14T13:43:41.330+0000 7f3b8fb6d7c0 -1 osd.1 86 log_to_monitors true 2024-04-14T13:43:41.569 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:41 smithi008 ceph-mon[67070]: pgmap v107: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.0 MiB/s wr, 621 op/s; 174/1623 objects degraded (10.721%) 2024-04-14T13:43:41.569 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:41 smithi008 ceph-mon[67070]: from='osd.1 [v2:172.21.15.8:6810/1834198135,v1:172.21.15.8:6811/1834198135]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T13:43:42.042 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:43:41 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1[75673]: 2024-04-14T13:43:41.900+0000 7f3b7cfbd700 -1 osd.1 86 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:43:42.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:41 smithi012 ceph-mon[53289]: pgmap v107: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.0 MiB/s wr, 621 op/s; 174/1623 objects degraded (10.721%) 2024-04-14T13:43:42.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:41 smithi012 ceph-mon[53289]: from='osd.1 [v2:172.21.15.8:6810/1834198135,v1:172.21.15.8:6811/1834198135]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T13:43:42.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 224899 11.14 MB/sec execute 376 sec latency 39.735 ms 2024-04-14T13:43:42.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:42 smithi008 ceph-mon[67070]: from='osd.1 [v2:172.21.15.8:6810/1834198135,v1:172.21.15.8:6811/1834198135]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-14T13:43:42.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:42 smithi008 ceph-mon[67070]: osdmap e89: 8 total, 7 up, 8 in 2024-04-14T13:43:42.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:42 smithi008 ceph-mon[67070]: from='osd.1 [v2:172.21.15.8:6810/1834198135,v1:172.21.15.8:6811/1834198135]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:43:42.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:42 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:42.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:42 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:42.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:42 smithi008 ceph-mon[67070]: from='osd.1 [v2:172.21.15.8:6810/1834198135,v1:172.21.15.8:6811/1834198135]' entity='osd.1' 2024-04-14T13:43:42.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:42 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 174/1623 objects degraded (10.721%), 23 pgs degraded (PG_DEGRADED) 2024-04-14T13:43:42.958 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:43:43.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:42 smithi012 ceph-mon[53289]: from='osd.1 [v2:172.21.15.8:6810/1834198135,v1:172.21.15.8:6811/1834198135]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-14T13:43:43.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:42 smithi012 ceph-mon[53289]: osdmap e89: 8 total, 7 up, 8 in 2024-04-14T13:43:43.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:42 smithi012 ceph-mon[53289]: from='osd.1 [v2:172.21.15.8:6810/1834198135,v1:172.21.15.8:6811/1834198135]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:43:43.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:42 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:43.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:42 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:43.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:42 smithi012 ceph-mon[53289]: from='osd.1 [v2:172.21.15.8:6810/1834198135,v1:172.21.15.8:6811/1834198135]' entity='osd.1' 2024-04-14T13:43:43.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:42 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 174/1623 objects degraded (10.721%), 23 pgs degraded (PG_DEGRADED) 2024-04-14T13:43:43.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 225177 11.12 MB/sec execute 377 sec latency 473.599 ms 2024-04-14T13:43:43.730 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:43:43.730 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (5m) 2s ago 14m 15.2M - 0.23.0 ba2b418f427c cfddb9a7f858 2024-04-14T13:43:43.730 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (97s) 2s ago 14m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 1e69eabdcc36 2024-04-14T13:43:43.730 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (90s) 85s ago 13m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 60c7a2992eb0 2024-04-14T13:43:43.730 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 2s ago 14m - - 2024-04-14T13:43:43.730 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (8m) 2s ago 8m 38.5M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:43:43.730 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (8m) 85s ago 8m 14.4M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:43:43.730 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (4m) 2s ago 15m 505M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:43:43.730 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (3m) 85s ago 13m 451M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:43:43.731 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (3m) 2s ago 16m 47.7M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 673dd6f4c8d8 2024-04-14T13:43:43.731 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (2m) 85s ago 13m 35.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 d498dec29c4f 2024-04-14T13:43:43.731 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (8m) 2s ago 8m 275M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:43:43.731 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (8m) 85s ago 8m 88.5M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:43:43.731 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (5m) 2s ago 14m 14.3M - 1.3.1 1dbe0e931976 646dfca70be4 2024-04-14T13:43:43.731 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (4m) 85s ago 13m 18.4M - 1.3.1 1dbe0e931976 0a60dcd4044c 2024-04-14T13:43:43.731 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (57s) 2s ago 13m 270M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 29997ccc66ac 2024-04-14T13:43:43.731 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (10s) 2s ago 12m 12.6M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 97df88f10bd6 2024-04-14T13:43:43.731 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (12m) 2s ago 12m 522M 1937M 17.2.0 e1d6a67b021e b1017f00578f 2024-04-14T13:43:43.731 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (11m) 2s ago 11m 401M 1937M 17.2.0 e1d6a67b021e 66fca4b191be 2024-04-14T13:43:43.731 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (11m) 85s ago 11m 614M 2705M 17.2.0 e1d6a67b021e 43b02cb64951 2024-04-14T13:43:43.731 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (11m) 85s ago 11m 430M 2705M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:43:43.731 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (10m) 85s ago 10m 341M 2705M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:43:43.731 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (10m) 85s ago 10m 552M 2705M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:43:43.731 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (3m) 2s ago 13m 69.6M - 2.33.4 514e6a882f6e 12f97f75d620 2024-04-14T13:43:43.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:43 smithi008 ceph-mon[67070]: pgmap v109: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.1 MiB/s wr, 498 op/s; 174/1623 objects degraded (10.721%) 2024-04-14T13:43:43.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:43 smithi008 ceph-mon[67070]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T13:43:43.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:43 smithi008 ceph-mon[67070]: osd.1 [v2:172.21.15.8:6810/1834198135,v1:172.21.15.8:6811/1834198135] boot 2024-04-14T13:43:43.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:43 smithi008 ceph-mon[67070]: osdmap e90: 8 total, 8 up, 8 in 2024-04-14T13:43:43.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:43 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:43:43.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:43 smithi008 ceph-mon[67070]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:43:43.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:43 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:43.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:43 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:44.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:43 smithi012 ceph-mon[53289]: pgmap v109: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.1 MiB/s wr, 498 op/s; 174/1623 objects degraded (10.721%) 2024-04-14T13:43:44.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:43 smithi012 ceph-mon[53289]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T13:43:44.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:43 smithi012 ceph-mon[53289]: osd.1 [v2:172.21.15.8:6810/1834198135,v1:172.21.15.8:6811/1834198135] boot 2024-04-14T13:43:44.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:43 smithi012 ceph-mon[53289]: osdmap e90: 8 total, 8 up, 8 in 2024-04-14T13:43:44.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:43 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:43:44.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:43 smithi012 ceph-mon[53289]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:43:44.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:43 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:44.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:43 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:44.297 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:43:44.297 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:43:44.297 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:43:44.297 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:43:44.297 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:43:44.297 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:43:44.297 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:43:44.297 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:43:44.297 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-14T13:43:44.298 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:43:44.298 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:43:44.298 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:43:44.298 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:43:44.298 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:43:44.298 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:43:44.298 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-04-14T13:43:44.298 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 6 2024-04-14T13:43:44.298 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:43:44.298 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:43:44.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 225237 11.10 MB/sec execute 378 sec latency 1190.277 ms 2024-04-14T13:43:44.754 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:43:44.755 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:43:44.755 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:43:44.755 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:43:44.755 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-14T13:43:44.755 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-04-14T13:43:44.755 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-14T13:43:44.755 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-04-14T13:43:44.755 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:43:44.755 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "8/23 daemons upgraded", 2024-04-14T13:43:44.755 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T13:43:44.755 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:43:44.755 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:43:45.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:44 smithi008 ceph-mon[67070]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:43:45.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:44 smithi008 ceph-mon[67070]: from='client.34262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:43:45.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:44 smithi008 ceph-mon[67070]: osdmap e91: 8 total, 8 up, 8 in 2024-04-14T13:43:45.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:44 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/659918883' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:43:45.185 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 126/1746 objects degraded (7.216%), 12 pgs degraded 2024-04-14T13:43:45.185 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:43:45.185 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-14T13:43:45.186 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 126/1746 objects degraded (7.216%), 12 pgs degraded 2024-04-14T13:43:45.186 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.4 is active+undersized+degraded, acting [0,7] 2024-04-14T13:43:45.186 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.9 is active+undersized+degraded, acting [7,3] 2024-04-14T13:43:45.186 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.a is active+undersized+degraded, acting [3,7] 2024-04-14T13:43:45.186 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.d is active+undersized+degraded, acting [4,3] 2024-04-14T13:43:45.186 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.15 is active+undersized+degraded, acting [5,0] 2024-04-14T13:43:45.186 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.4 is active+undersized+degraded, acting [2,5] 2024-04-14T13:43:45.186 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.6 is active+undersized+degraded, acting [0,4] 2024-04-14T13:43:45.186 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.8 is active+recovering+degraded, acting [3,1,7] 2024-04-14T13:43:45.186 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.a is active+recovery_wait+degraded, acting [6,4,1] 2024-04-14T13:43:45.186 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.16 is active+recovery_wait+degraded, acting [5,7,1] 2024-04-14T13:43:45.186 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.18 is active+recovery_wait+degraded, acting [3,0,1] 2024-04-14T13:43:45.186 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1c is active+recovery_wait+degraded, acting [5,4,1] 2024-04-14T13:43:45.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:44 smithi012 ceph-mon[53289]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:43:45.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:44 smithi012 ceph-mon[53289]: from='client.34262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:43:45.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:44 smithi012 ceph-mon[53289]: osdmap e91: 8 total, 8 up, 8 in 2024-04-14T13:43:45.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:44 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/659918883' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:43:45.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 225306 11.07 MB/sec execute 379 sec latency 1251.767 ms 2024-04-14T13:43:45.948 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:45 smithi008 ceph-mon[67070]: pgmap v112: 97 pgs: 7 remapped+peering, 1 active+recovering+degraded, 7 active+undersized+degraded, 4 active+recovery_wait+degraded, 7 active+undersized, 71 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 14 KiB/s rd, 7.4 MiB/s wr, 801 op/s; 126/1746 objects degraded (7.216%); 0 B/s, 1 objects/s recovering 2024-04-14T13:43:45.949 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:45 smithi008 ceph-mon[67070]: from='client.44207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:43:45.949 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:45 smithi008 ceph-mon[67070]: osdmap e92: 8 total, 8 up, 8 in 2024-04-14T13:43:45.949 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:45 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/1134460780' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:43:46.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:45 smithi012 ceph-mon[53289]: pgmap v112: 97 pgs: 7 remapped+peering, 1 active+recovering+degraded, 7 active+undersized+degraded, 4 active+recovery_wait+degraded, 7 active+undersized, 71 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 14 KiB/s rd, 7.4 MiB/s wr, 801 op/s; 126/1746 objects degraded (7.216%); 0 B/s, 1 objects/s recovering 2024-04-14T13:43:46.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:45 smithi012 ceph-mon[53289]: from='client.44207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:43:46.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:45 smithi012 ceph-mon[53289]: osdmap e92: 8 total, 8 up, 8 in 2024-04-14T13:43:46.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:45 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/1134460780' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:43:46.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 225550 11.06 MB/sec execute 380 sec latency 1324.780 ms 2024-04-14T13:43:46.966 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:46 smithi008 ceph-mon[67070]: osdmap e93: 8 total, 8 up, 8 in 2024-04-14T13:43:47.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:46 smithi012 ceph-mon[53289]: osdmap e93: 8 total, 8 up, 8 in 2024-04-14T13:43:47.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 225907 11.06 MB/sec execute 381 sec latency 344.441 ms 2024-04-14T13:43:47.982 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:47 smithi008 ceph-mon[67070]: pgmap v115: 97 pgs: 8 remapped+peering, 1 active+recovering+degraded, 6 active+undersized+degraded, 4 active+recovery_wait+degraded, 5 active+undersized, 73 active+clean; 349 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 7.8 MiB/s wr, 727 op/s; 112/1755 objects degraded (6.382%); 0 B/s, 2 objects/s recovering 2024-04-14T13:43:47.982 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:47 smithi008 ceph-mon[67070]: osdmap e94: 8 total, 8 up, 8 in 2024-04-14T13:43:47.983 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:47 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 112/1755 objects degraded (6.382%), 11 pgs degraded (PG_DEGRADED) 2024-04-14T13:43:47.983 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:47 smithi008 ceph-mon[67070]: osdmap e95: 8 total, 8 up, 8 in 2024-04-14T13:43:48.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:47 smithi012 ceph-mon[53289]: pgmap v115: 97 pgs: 8 remapped+peering, 1 active+recovering+degraded, 6 active+undersized+degraded, 4 active+recovery_wait+degraded, 5 active+undersized, 73 active+clean; 349 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 7.8 MiB/s wr, 727 op/s; 112/1755 objects degraded (6.382%); 0 B/s, 2 objects/s recovering 2024-04-14T13:43:48.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:47 smithi012 ceph-mon[53289]: osdmap e94: 8 total, 8 up, 8 in 2024-04-14T13:43:48.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:47 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 112/1755 objects degraded (6.382%), 11 pgs degraded (PG_DEGRADED) 2024-04-14T13:43:48.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:47 smithi012 ceph-mon[53289]: osdmap e95: 8 total, 8 up, 8 in 2024-04-14T13:43:48.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 226195 11.05 MB/sec execute 382 sec latency 1013.513 ms 2024-04-14T13:43:49.330 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:49.330 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:43:49.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:49 smithi008 ceph-mon[67070]: osdmap e96: 8 total, 8 up, 8 in 2024-04-14T13:43:49.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:49 smithi008 ceph-mon[67070]: pgmap v119: 97 pgs: 1 peering, 84 active+clean, 4 active+recovery_wait+degraded, 1 active+recovering+degraded, 7 remapped+peering; 351 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 883 KiB/s wr, 169 op/s; 81/1752 objects degraded (4.623%); 44 KiB/s, 6 objects/s recovering 2024-04-14T13:43:49.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:49.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:49.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:43:49.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:43:49.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:49 smithi008 ceph-mon[67070]: pgmap v120: 97 pgs: 1 peering, 84 active+clean, 4 active+recovery_wait+degraded, 1 active+recovering+degraded, 7 remapped+peering; 351 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.1 MiB/s wr, 220 op/s; 81/1752 objects degraded (4.623%); 57 KiB/s, 8 objects/s recovering 2024-04-14T13:43:49.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:49 smithi008 ceph-mon[67070]: pgmap v121: 97 pgs: 1 peering, 84 active+clean, 4 active+recovery_wait+degraded, 1 active+recovering+degraded, 7 remapped+peering; 351 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.4 MiB/s wr, 274 op/s; 81/1752 objects degraded (4.623%); 71 KiB/s, 10 objects/s recovering 2024-04-14T13:43:49.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:49.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:49.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:43:49.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:43:49.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:43:49.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:43:49.332 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T13:43:49.332 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:49 smithi008 ceph-mon[67070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T13:43:49.332 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:49 smithi008 ceph-mon[67070]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-04-14T13:43:49.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 226526 11.04 MB/sec execute 383 sec latency 293.586 ms 2024-04-14T13:43:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:43:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:49 smithi012 ceph-mon[53289]: osdmap e96: 8 total, 8 up, 8 in 2024-04-14T13:43:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:49 smithi012 ceph-mon[53289]: pgmap v119: 97 pgs: 1 peering, 84 active+clean, 4 active+recovery_wait+degraded, 1 active+recovering+degraded, 7 remapped+peering; 351 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 883 KiB/s wr, 169 op/s; 81/1752 objects degraded (4.623%); 44 KiB/s, 6 objects/s recovering 2024-04-14T13:43:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:43:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:43:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:49 smithi012 ceph-mon[53289]: pgmap v120: 97 pgs: 1 peering, 84 active+clean, 4 active+recovery_wait+degraded, 1 active+recovering+degraded, 7 remapped+peering; 351 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.1 MiB/s wr, 220 op/s; 81/1752 objects degraded (4.623%); 57 KiB/s, 8 objects/s recovering 2024-04-14T13:43:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:49 smithi012 ceph-mon[53289]: pgmap v121: 97 pgs: 1 peering, 84 active+clean, 4 active+recovery_wait+degraded, 1 active+recovering+degraded, 7 remapped+peering; 351 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.4 MiB/s wr, 274 op/s; 81/1752 objects degraded (4.623%); 71 KiB/s, 10 objects/s recovering 2024-04-14T13:43:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:49.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:43:49.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:43:49.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:43:49.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:43:49.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T13:43:49.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:49 smithi012 ceph-mon[53289]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T13:43:49.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:49 smithi012 ceph-mon[53289]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-04-14T13:43:50.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 226770 11.03 MB/sec execute 384 sec latency 56.343 ms 2024-04-14T13:43:50.378 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:50 smithi008 ceph-mon[67070]: osdmap e97: 8 total, 8 up, 8 in 2024-04-14T13:43:50.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:50 smithi012 ceph-mon[53289]: osdmap e97: 8 total, 8 up, 8 in 2024-04-14T13:43:51.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 226950 11.01 MB/sec execute 385 sec latency 58.557 ms 2024-04-14T13:43:51.408 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:51 smithi008 ceph-mon[67070]: pgmap v123: 97 pgs: 1 peering, 94 active+clean, 2 remapped+peering; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 3.4 MiB/s wr, 272 op/s; 2.4 MiB/s, 53 objects/s recovering 2024-04-14T13:43:51.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:51 smithi012 ceph-mon[53289]: pgmap v123: 97 pgs: 1 peering, 94 active+clean, 2 remapped+peering; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 3.4 MiB/s wr, 272 op/s; 2.4 MiB/s, 53 objects/s recovering 2024-04-14T13:43:52.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 227280 11.01 MB/sec execute 386 sec latency 88.355 ms 2024-04-14T13:43:52.426 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:52 smithi008 ceph-mon[67070]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 81/1752 objects degraded (4.623%), 5 pgs degraded) 2024-04-14T13:43:52.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:52 smithi012 ceph-mon[53289]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 81/1752 objects degraded (4.623%), 5 pgs degraded) 2024-04-14T13:43:53.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 227628 11.01 MB/sec execute 387 sec latency 61.297 ms 2024-04-14T13:43:53.452 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:53 smithi008 ceph-mon[67070]: pgmap v124: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.5 MiB/s wr, 328 op/s; 2.8 MiB/s, 73 keys/s, 51 objects/s recovering 2024-04-14T13:43:53.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:53 smithi012 ceph-mon[53289]: pgmap v124: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.5 MiB/s wr, 328 op/s; 2.8 MiB/s, 73 keys/s, 51 objects/s recovering 2024-04-14T13:43:54.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 227957 11.01 MB/sec execute 388 sec latency 60.519 ms 2024-04-14T13:43:55.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 228281 11.01 MB/sec execute 389 sec latency 35.726 ms 2024-04-14T13:43:55.735 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:55 smithi008 ceph-mon[67070]: pgmap v125: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.6 MiB/s wr, 536 op/s; 2.0 MiB/s, 52 keys/s, 36 objects/s recovering 2024-04-14T13:43:56.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:55 smithi012 ceph-mon[53289]: pgmap v125: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.6 MiB/s wr, 536 op/s; 2.0 MiB/s, 52 keys/s, 36 objects/s recovering 2024-04-14T13:43:56.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 228567 10.99 MB/sec execute 390 sec latency 51.098 ms 2024-04-14T13:43:57.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 228833 10.98 MB/sec execute 391 sec latency 64.123 ms 2024-04-14T13:43:57.757 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:57 smithi008 ceph-mon[67070]: pgmap v126: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.9 MiB/s wr, 441 op/s; 1.5 MiB/s, 40 keys/s, 28 objects/s recovering 2024-04-14T13:43:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:57 smithi012 ceph-mon[53289]: pgmap v126: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.9 MiB/s wr, 441 op/s; 1.5 MiB/s, 40 keys/s, 28 objects/s recovering 2024-04-14T13:43:58.357 INFO:teuthology.orchestra.run.smithi008.stdout: 5 229001 10.96 MB/sec execute 392 sec latency 78.394 ms 2024-04-14T13:43:59.326 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:59 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:59.326 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:43:59 smithi008 ceph-mon[67070]: pgmap v127: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.9 MiB/s wr, 353 op/s; 1.2 MiB/s, 32 keys/s, 22 objects/s recovering 2024-04-14T13:43:59.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 229245 10.95 MB/sec execute 393 sec latency 63.943 ms 2024-04-14T13:43:59.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:59 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:43:59.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:43:59 smithi012 ceph-mon[53289]: pgmap v127: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.9 MiB/s wr, 353 op/s; 1.2 MiB/s, 32 keys/s, 22 objects/s recovering 2024-04-14T13:44:00.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 229590 10.95 MB/sec execute 394 sec latency 49.763 ms 2024-04-14T13:44:01.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 230014 10.94 MB/sec execute 395 sec latency 37.715 ms 2024-04-14T13:44:01.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:01 smithi008 ceph-mon[67070]: pgmap v128: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.3 MiB/s wr, 416 op/s; 383 KiB/s, 28 keys/s, 6 objects/s recovering 2024-04-14T13:44:02.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:01 smithi012 ceph-mon[53289]: pgmap v128: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.3 MiB/s wr, 416 op/s; 383 KiB/s, 28 keys/s, 6 objects/s recovering 2024-04-14T13:44:02.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 230365 10.94 MB/sec execute 396 sec latency 41.698 ms 2024-04-14T13:44:03.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 230724 10.94 MB/sec execute 397 sec latency 38.115 ms 2024-04-14T13:44:03.901 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:03 smithi008 ceph-mon[67070]: pgmap v129: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.1 MiB/s wr, 438 op/s; 341 KiB/s, 26 keys/s, 3 objects/s recovering 2024-04-14T13:44:03.902 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:03.902 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:44:04.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:03 smithi012 ceph-mon[53289]: pgmap v129: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.1 MiB/s wr, 438 op/s; 341 KiB/s, 26 keys/s, 3 objects/s recovering 2024-04-14T13:44:04.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:04.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:44:04.354 INFO:teuthology.orchestra.run.smithi008.stdout: 5 231132 10.94 MB/sec execute 398 sec latency 44.182 ms 2024-04-14T13:44:04.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:04 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T13:44:04.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:04 smithi008 ceph-mon[67070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T13:44:04.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:04 smithi008 ceph-mon[67070]: Upgrade: osd.2 is safe to restart 2024-04-14T13:44:05.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:04 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T13:44:05.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:04 smithi012 ceph-mon[53289]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T13:44:05.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:04 smithi012 ceph-mon[53289]: Upgrade: osd.2 is safe to restart 2024-04-14T13:44:05.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 231422 10.94 MB/sec execute 399 sec latency 51.802 ms 2024-04-14T13:44:05.943 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:05 smithi008 ceph-mon[67070]: pgmap v130: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.0 MiB/s wr, 470 op/s 2024-04-14T13:44:06.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:05 smithi012 ceph-mon[53289]: pgmap v130: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.0 MiB/s wr, 470 op/s 2024-04-14T13:44:06.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 231746 10.94 MB/sec execute 400 sec latency 32.785 ms 2024-04-14T13:44:07.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 231999 10.92 MB/sec execute 401 sec latency 69.687 ms 2024-04-14T13:44:07.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:07 smithi008 ceph-mon[67070]: Upgrade: Updating osd.2 2024-04-14T13:44:07.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:07 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:07.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:07 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T13:44:07.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:07 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:44:07.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:07 smithi008 ceph-mon[67070]: Deploying daemon osd.2 on smithi008 2024-04-14T13:44:07.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:07 smithi008 ceph-mon[67070]: pgmap v131: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.5 MiB/s wr, 471 op/s 2024-04-14T13:44:08.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:07 smithi012 ceph-mon[53289]: Upgrade: Updating osd.2 2024-04-14T13:44:08.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:07 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:08.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:07 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T13:44:08.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:07 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:44:08.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:07 smithi012 ceph-mon[53289]: Deploying daemon osd.2 on smithi008 2024-04-14T13:44:08.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:07 smithi012 ceph-mon[53289]: pgmap v131: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.5 MiB/s wr, 471 op/s 2024-04-14T13:44:08.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 232321 10.91 MB/sec execute 402 sec latency 51.197 ms 2024-04-14T13:44:09.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 232560 10.90 MB/sec execute 403 sec latency 56.187 ms 2024-04-14T13:44:09.776 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:09 smithi008 ceph-mon[67070]: pgmap v132: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.1 MiB/s wr, 453 op/s 2024-04-14T13:44:10.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:09 smithi012 ceph-mon[53289]: pgmap v132: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.1 MiB/s wr, 453 op/s 2024-04-14T13:44:10.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 232850 10.89 MB/sec execute 404 sec latency 34.792 ms 2024-04-14T13:44:11.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 233208 10.89 MB/sec execute 405 sec latency 47.712 ms 2024-04-14T13:44:11.817 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:11 smithi008 ceph-mon[67070]: pgmap v133: 97 pgs: 97 active+clean; 349 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.6 MiB/s wr, 574 op/s 2024-04-14T13:44:12.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:11 smithi012 ceph-mon[53289]: pgmap v133: 97 pgs: 97 active+clean; 349 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.6 MiB/s wr, 574 op/s 2024-04-14T13:44:12.328 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:12 smithi008 systemd[1]: Stopping Ceph osd.2 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:44:12.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 233579 10.88 MB/sec execute 406 sec latency 54.948 ms 2024-04-14T13:44:12.582 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:12 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2[41952]: 2024-04-14T13:44:12.503+0000 7f862287f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:44:12.582 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:12 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2[41952]: 2024-04-14T13:44:12.503+0000 7f862287f700 -1 osd.2 97 *** Got signal Terminated *** 2024-04-14T13:44:12.582 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:12 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2[41952]: 2024-04-14T13:44:12.503+0000 7f862287f700 -1 osd.2 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T13:44:12.833 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:12 smithi008 ceph-mon[67070]: osd.2 marked itself down and dead 2024-04-14T13:44:13.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:12 smithi012 ceph-mon[53289]: osd.2 marked itself down and dead 2024-04-14T13:44:13.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 233787 10.87 MB/sec execute 407 sec latency 634.401 ms 2024-04-14T13:44:13.603 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:13 smithi008 podman[77796]: 2024-04-14 13:44:13.274624207 +0000 UTC m=+0.903306774 container died b1017f00578fb73ffbf01d47c89e9490f68bda627b18f048e3d49a10316e70e6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, architecture=x86_64, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.19.8, RELEASE=HEAD, release=754, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, name=centos-stream, vendor=Red Hat, Inc., version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, vcs-type=git, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-14T13:44:13.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:13 smithi008 ceph-mon[67070]: pgmap v134: 97 pgs: 97 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.1 MiB/s wr, 414 op/s 2024-04-14T13:44:13.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:13 smithi008 ceph-mon[67070]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T13:44:13.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:13 smithi008 ceph-mon[67070]: osdmap e98: 8 total, 7 up, 8 in 2024-04-14T13:44:13.865 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:13 smithi008 podman[77796]: 2024-04-14 13:44:13.802417417 +0000 UTC m=+1.431099978 container cleanup b1017f00578fb73ffbf01d47c89e9490f68bda627b18f048e3d49a10316e70e6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2, vcs-type=git, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, name=centos-stream, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, release=754, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-14T13:44:13.865 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:13 smithi008 bash[77796]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2 2024-04-14T13:44:14.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:13 smithi012 ceph-mon[53289]: pgmap v134: 97 pgs: 97 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.1 MiB/s wr, 414 op/s 2024-04-14T13:44:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:13 smithi012 ceph-mon[53289]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T13:44:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:13 smithi012 ceph-mon[53289]: osdmap e98: 8 total, 7 up, 8 in 2024-04-14T13:44:14.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 233928 10.85 MB/sec execute 408 sec latency 1051.089 ms 2024-04-14T13:44:14.890 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:14 smithi008 ceph-mon[67070]: osdmap e99: 8 total, 7 up, 8 in 2024-04-14T13:44:14.891 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:14 smithi008 podman[77815]: 2024-04-14 13:44:14.646923451 +0000 UTC m=+1.372894442 container remove b1017f00578fb73ffbf01d47c89e9490f68bda627b18f048e3d49a10316e70e6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, version=8, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, vcs-type=git, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, release=754, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, architecture=x86_64, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True) 2024-04-14T13:44:15.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:14 smithi012 ceph-mon[53289]: osdmap e99: 8 total, 7 up, 8 in 2024-04-14T13:44:15.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 234292 10.86 MB/sec execute 409 sec latency 37.034 ms 2024-04-14T13:44:15.401 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:15 smithi008 podman[77897]: 2024-04-14 13:44:15.064618043 +0000 UTC m=+0.025106365 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:44:15.401 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:15 smithi008 podman[77897]: 2024-04-14 13:44:15.18963959 +0000 UTC m=+0.150127944 container create 6953a554ac99fe9f69e5b6bdfe1d28970c97f81931b4fea806c9e41a05b5e862 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-14T13:44:15.652 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:15 smithi008 podman[77897]: 2024-04-14 13:44:15.609748248 +0000 UTC m=+0.570236569 container init 6953a554ac99fe9f69e5b6bdfe1d28970c97f81931b4fea806c9e41a05b5e862 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-04-14T13:44:15.652 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:15 smithi008 podman[77897]: 2024-04-14 13:44:15.61999389 +0000 UTC m=+0.580482203 container start 6953a554ac99fe9f69e5b6bdfe1d28970c97f81931b4fea806c9e41a05b5e862 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.schema-version=1.0) 2024-04-14T13:44:15.711 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:44:15.903 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:15 smithi008 podman[77897]: 2024-04-14 13:44:15.778431436 +0000 UTC m=+0.738919742 container attach 6953a554ac99fe9f69e5b6bdfe1d28970c97f81931b4fea806c9e41a05b5e862 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-14T13:44:15.903 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:15 smithi008 ceph-mon[67070]: pgmap v137: 97 pgs: 2 active+undersized, 15 active+undersized+degraded, 2 stale+active+clean, 78 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.3 MiB/s wr, 414 op/s; 93/1551 objects degraded (5.996%) 2024-04-14T13:44:15.903 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:15 smithi008 ceph-mon[67070]: Health check failed: Degraded data redundancy: 93/1551 objects degraded (5.996%), 15 pgs degraded (PG_DEGRADED) 2024-04-14T13:44:16.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:15 smithi012 ceph-mon[53289]: pgmap v137: 97 pgs: 2 active+undersized, 15 active+undersized+degraded, 2 stale+active+clean, 78 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.3 MiB/s wr, 414 op/s; 93/1551 objects degraded (5.996%) 2024-04-14T13:44:16.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:15 smithi012 ceph-mon[53289]: Health check failed: Degraded data redundancy: 93/1551 objects degraded (5.996%), 15 pgs degraded (PG_DEGRADED) 2024-04-14T13:44:16.154 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:16 smithi008 podman[77897]: 2024-04-14 13:44:16.096958645 +0000 UTC m=+1.057446956 container died 6953a554ac99fe9f69e5b6bdfe1d28970c97f81931b4fea806c9e41a05b5e862 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T13:44:16.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 234798 10.86 MB/sec execute 410 sec latency 27.429 ms 2024-04-14T13:44:16.662 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:44:16.662 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (6m) 35s ago 15m 15.2M - 0.23.0 ba2b418f427c cfddb9a7f858 2024-04-14T13:44:16.662 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (2m) 35s ago 15m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 1e69eabdcc36 2024-04-14T13:44:16.662 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (2m) 118s ago 14m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 60c7a2992eb0 2024-04-14T13:44:16.662 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 35s ago 14m - - 2024-04-14T13:44:16.662 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (9m) 35s ago 9m 38.5M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:44:16.663 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (9m) 118s ago 9m 14.4M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:44:16.663 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (5m) 35s ago 16m 505M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:44:16.663 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (4m) 118s ago 14m 451M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:44:16.663 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (3m) 35s ago 16m 47.7M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 673dd6f4c8d8 2024-04-14T13:44:16.663 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (3m) 118s ago 14m 35.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 d498dec29c4f 2024-04-14T13:44:16.663 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (9m) 35s ago 9m 275M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:44:16.663 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (8m) 118s ago 8m 88.5M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:44:16.663 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (5m) 35s ago 14m 14.3M - 1.3.1 1dbe0e931976 646dfca70be4 2024-04-14T13:44:16.663 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (5m) 118s ago 14m 18.4M - 1.3.1 1dbe0e931976 0a60dcd4044c 2024-04-14T13:44:16.663 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (90s) 35s ago 13m 270M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 29997ccc66ac 2024-04-14T13:44:16.663 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (43s) 35s ago 13m 12.6M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 97df88f10bd6 2024-04-14T13:44:16.663 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (12m) 35s ago 12m 522M 1937M 17.2.0 e1d6a67b021e b1017f00578f 2024-04-14T13:44:16.663 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (12m) 35s ago 12m 401M 1937M 17.2.0 e1d6a67b021e 66fca4b191be 2024-04-14T13:44:16.663 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (11m) 118s ago 11m 614M 2705M 17.2.0 e1d6a67b021e 43b02cb64951 2024-04-14T13:44:16.663 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (11m) 118s ago 11m 430M 2705M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:44:16.664 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (11m) 118s ago 11m 341M 2705M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:44:16.664 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (10m) 118s ago 10m 552M 2705M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:44:16.664 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (4m) 35s ago 13m 69.6M - 2.33.4 514e6a882f6e 12f97f75d620 2024-04-14T13:44:16.916 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:16 smithi008 ceph-mon[67070]: from='client.34276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:44:17.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:16 smithi012 ceph-mon[53289]: from='client.34276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:44:17.170 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:17 smithi008 podman[77969]: 2024-04-14 13:44:17.018771911 +0000 UTC m=+0.912796518 container remove 6953a554ac99fe9f69e5b6bdfe1d28970c97f81931b4fea806c9e41a05b5e862 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-deactivate, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-14T13:44:17.170 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:17 smithi008 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-14T13:44:17.170 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:17 smithi008 systemd[1]: Stopped Ceph osd.2 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:44:17.170 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:17 smithi008 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.2.service: Consumed 3min 916ms CPU time. 2024-04-14T13:44:17.320 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:44:17.320 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:44:17.321 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:44:17.321 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:44:17.321 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:44:17.321 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:44:17.321 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:44:17.321 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:44:17.321 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-14T13:44:17.321 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:44:17.321 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:44:17.321 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:44:17.321 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:44:17.321 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:44:17.321 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:44:17.322 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-14T13:44:17.322 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 6 2024-04-14T13:44:17.322 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:44:17.322 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:44:17.358 INFO:teuthology.orchestra.run.smithi008.stdout: 5 235141 10.86 MB/sec execute 411 sec latency 50.777 ms 2024-04-14T13:44:17.425 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:17 smithi008 systemd[1]: Starting Ceph osd.2 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:44:17.676 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:17 smithi008 podman[78132]: 2024-04-14 13:44:17.564409228 +0000 UTC m=+0.024531129 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:44:17.849 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:44:17.849 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:44:17.849 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:44:17.849 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:44:17.849 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-14T13:44:17.849 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-04-14T13:44:17.849 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-14T13:44:17.849 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-04-14T13:44:17.849 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:44:17.849 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "8/23 daemons upgraded", 2024-04-14T13:44:17.849 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T13:44:17.849 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:44:17.849 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:44:17.929 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:17 smithi008 podman[78132]: 2024-04-14 13:44:17.720611371 +0000 UTC m=+0.180733270 container create 25ad71df7af1cdb1811d10744153b907e3dcc82d6633a99989af212637bbeaf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-14T13:44:17.929 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:17 smithi008 ceph-mon[67070]: from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:44:17.929 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:17 smithi008 ceph-mon[67070]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:44:17.929 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:17 smithi008 ceph-mon[67070]: pgmap v138: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.0 MiB/s wr, 448 op/s; 134/1575 objects degraded (8.508%) 2024-04-14T13:44:17.929 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:17 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/2892473634' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:44:18.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:17 smithi012 ceph-mon[53289]: from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:44:18.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:17 smithi012 ceph-mon[53289]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:44:18.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:17 smithi012 ceph-mon[53289]: pgmap v138: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.0 MiB/s wr, 448 op/s; 134/1575 objects degraded (8.508%) 2024-04-14T13:44:18.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:17 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/2892473634' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:44:18.213 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:18 smithi008 podman[78132]: 2024-04-14 13:44:18.116907298 +0000 UTC m=+0.577029201 container init 25ad71df7af1cdb1811d10744153b907e3dcc82d6633a99989af212637bbeaf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:44:18.213 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:18 smithi008 podman[78132]: 2024-04-14 13:44:18.139745039 +0000 UTC m=+0.599866932 container start 25ad71df7af1cdb1811d10744153b907e3dcc82d6633a99989af212637bbeaf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-activate, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-14T13:44:18.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 235489 10.86 MB/sec execute 412 sec latency 61.981 ms 2024-04-14T13:44:18.464 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:18 smithi008 podman[78132]: 2024-04-14 13:44:18.309074912 +0000 UTC m=+0.769196807 container attach 25ad71df7af1cdb1811d10744153b907e3dcc82d6633a99989af212637bbeaf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-14T13:44:18.570 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 134/1575 objects degraded (8.508%), 19 pgs degraded 2024-04-14T13:44:18.571 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:44:18.571 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-14T13:44:18.571 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-14T13:44:18.571 INFO:teuthology.orchestra.run.smithi008.stdout: osd.2 (root=default,host=smithi008) is down 2024-04-14T13:44:18.571 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 134/1575 objects degraded (8.508%), 19 pgs degraded 2024-04-14T13:44:18.571 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1 is active+undersized+degraded, acting [3,0] 2024-04-14T13:44:18.571 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.3 is active+undersized+degraded, acting [5,7] 2024-04-14T13:44:18.571 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.7 is active+undersized+degraded, acting [6,7] 2024-04-14T13:44:18.571 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.c is active+undersized+degraded, acting [5,0] 2024-04-14T13:44:18.571 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.e is active+undersized+degraded, acting [3,4] 2024-04-14T13:44:18.571 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.10 is active+undersized+degraded, acting [1,0] 2024-04-14T13:44:18.571 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.16 is active+undersized+degraded, acting [5,6] 2024-04-14T13:44:18.571 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.17 is active+undersized+degraded, acting [6,5] 2024-04-14T13:44:18.571 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1c is active+undersized+degraded, acting [7,5] 2024-04-14T13:44:18.571 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.0 is active+undersized+degraded, acting [1,6] 2024-04-14T13:44:18.572 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.4 is active+undersized+degraded, acting [1,5] 2024-04-14T13:44:18.572 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.5 is active+undersized+degraded, acting [5,3] 2024-04-14T13:44:18.572 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.9 is active+undersized+degraded, acting [4,7] 2024-04-14T13:44:18.572 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.13 is active+undersized+degraded, acting [7,4] 2024-04-14T13:44:18.572 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1a is active+undersized+degraded, acting [4,1] 2024-04-14T13:44:18.572 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1e is active+undersized+degraded, acting [3,6] 2024-04-14T13:44:18.572 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1f is active+undersized+degraded, acting [0,5] 2024-04-14T13:44:18.572 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.a is active+undersized+degraded, acting [6,7] 2024-04-14T13:44:18.572 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.19 is active+undersized+degraded, acting [3,0] 2024-04-14T13:44:18.750 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:18 smithi008 ceph-mon[67070]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:44:18.751 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:18 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:18.751 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:18 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:44:18.751 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:18 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/1728546324' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:44:18.969 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:18 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-activate[78150]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-14T13:44:18.970 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:18 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-activate[78150]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-14T13:44:18.970 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:18 smithi008 bash[78132]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-14T13:44:18.970 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:18 smithi008 bash[78132]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-14T13:44:18.970 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:18 smithi008 bash[78132]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-14T13:44:18.970 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:18 smithi008 bash[78132]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-14T13:44:18.970 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:18 smithi008 bash[78132]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-14T13:44:18.970 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:18 smithi008 bash[78132]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-14T13:44:18.970 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:18 smithi008 bash[78132]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-14T13:44:18.970 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:18 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-activate[78150]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-14T13:44:18.970 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:18 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-activate[78150]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-14T13:44:18.970 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:18 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-activate[78150]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-14T13:44:18.970 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:18 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-activate[78150]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-14T13:44:18.970 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:18 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-activate[78150]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-14T13:44:18.971 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:18 smithi008 podman[78132]: 2024-04-14 13:44:18.882042448 +0000 UTC m=+1.342164338 container died 25ad71df7af1cdb1811d10744153b907e3dcc82d6633a99989af212637bbeaf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-activate, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T13:44:19.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:18 smithi012 ceph-mon[53289]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:44:19.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:18 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:19.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:18 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:44:19.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:18 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/1728546324' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:44:19.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 235914 10.86 MB/sec execute 413 sec latency 38.202 ms 2024-04-14T13:44:19.977 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:19 smithi008 ceph-mon[67070]: pgmap v139: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.8 MiB/s wr, 266 op/s; 134/1575 objects degraded (8.508%) 2024-04-14T13:44:20.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:19 smithi012 ceph-mon[53289]: pgmap v139: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.8 MiB/s wr, 266 op/s; 134/1575 objects degraded (8.508%) 2024-04-14T13:44:20.228 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:19 smithi008 podman[78305]: 2024-04-14 13:44:19.992285061 +0000 UTC m=+1.104144846 container remove 25ad71df7af1cdb1811d10744153b907e3dcc82d6633a99989af212637bbeaf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-activate, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:44:20.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 236254 10.84 MB/sec execute 414 sec latency 47.622 ms 2024-04-14T13:44:20.492 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:20 smithi008 podman[78361]: 2024-04-14 13:44:20.173978737 +0000 UTC m=+0.039054919 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:44:20.492 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:20 smithi008 podman[78361]: 2024-04-14 13:44:20.385797297 +0000 UTC m=+0.250873478 container create 9325b527c16c31beb9a724107e170aabcb3f4fc4743f8f604fcf465c3d19a5e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-14T13:44:20.997 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:20 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 146/1701 objects degraded (8.583%), 19 pgs degraded (PG_DEGRADED) 2024-04-14T13:44:21.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:20 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 146/1701 objects degraded (8.583%), 19 pgs degraded (PG_DEGRADED) 2024-04-14T13:44:21.249 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:21 smithi008 podman[78361]: 2024-04-14 13:44:21.151126327 +0000 UTC m=+1.016202520 container init 9325b527c16c31beb9a724107e170aabcb3f4fc4743f8f604fcf465c3d19a5e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0) 2024-04-14T13:44:21.249 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:21 smithi008 podman[78361]: 2024-04-14 13:44:21.163555855 +0000 UTC m=+1.028632043 container start 9325b527c16c31beb9a724107e170aabcb3f4fc4743f8f604fcf465c3d19a5e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_CLEAN=True, ceph=True) 2024-04-14T13:44:21.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 236632 10.85 MB/sec execute 415 sec latency 46.572 ms 2024-04-14T13:44:21.500 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:21 smithi008 bash[78361]: 9325b527c16c31beb9a724107e170aabcb3f4fc4743f8f604fcf465c3d19a5e7 2024-04-14T13:44:21.500 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:21 smithi008 systemd[1]: Started Ceph osd.2 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:44:22.012 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:21 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2[78375]: 2024-04-14T13:44:21.757+0000 7f7ba381d7c0 -1 Falling back to public interface 2024-04-14T13:44:22.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:21 smithi008 ceph-mon[67070]: pgmap v140: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.9 MiB/s wr, 593 op/s; 146/1701 objects degraded (8.583%) 2024-04-14T13:44:22.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:21 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:22.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:21 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:22.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:21 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:44:22.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:21 smithi012 ceph-mon[53289]: pgmap v140: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.9 MiB/s wr, 593 op/s; 146/1701 objects degraded (8.583%) 2024-04-14T13:44:22.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:21 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:22.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:21 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:22.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:21 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:44:22.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 237025 10.84 MB/sec execute 416 sec latency 48.646 ms 2024-04-14T13:44:23.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 237441 10.84 MB/sec execute 417 sec latency 39.463 ms 2024-04-14T13:44:24.037 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:23 smithi008 ceph-mon[67070]: pgmap v141: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.1 MiB/s wr, 539 op/s; 144/1686 objects degraded (8.541%) 2024-04-14T13:44:24.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:23 smithi012 ceph-mon[53289]: pgmap v141: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.1 MiB/s wr, 539 op/s; 144/1686 objects degraded (8.541%) 2024-04-14T13:44:24.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 237697 10.84 MB/sec execute 418 sec latency 36.335 ms 2024-04-14T13:44:25.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 238005 10.84 MB/sec execute 419 sec latency 54.397 ms 2024-04-14T13:44:25.824 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:25 smithi008 ceph-mon[67070]: pgmap v142: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 344 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.2 MiB/s wr, 617 op/s; 143/1662 objects degraded (8.604%) 2024-04-14T13:44:26.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:25 smithi012 ceph-mon[53289]: pgmap v142: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 344 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.2 MiB/s wr, 617 op/s; 143/1662 objects degraded (8.604%) 2024-04-14T13:44:26.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 238308 10.83 MB/sec execute 420 sec latency 45.341 ms 2024-04-14T13:44:26.645 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:26 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2[78375]: 2024-04-14T13:44:26.591+0000 7f7ba381d7c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-04-14T13:44:26.897 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:26 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 148/1662 objects degraded (8.905%), 19 pgs degraded (PG_DEGRADED) 2024-04-14T13:44:27.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:26 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 148/1662 objects degraded (8.905%), 19 pgs degraded (PG_DEGRADED) 2024-04-14T13:44:27.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 238544 10.82 MB/sec execute 421 sec latency 98.975 ms 2024-04-14T13:44:27.918 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:27 smithi008 ceph-mon[67070]: pgmap v143: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.0 MiB/s wr, 485 op/s; 148/1662 objects degraded (8.905%) 2024-04-14T13:44:27.918 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:27 smithi008 ceph-mon[67070]: from='osd.2 [v2:172.21.15.8:6818/2947889393,v1:172.21.15.8:6819/2947889393]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T13:44:27.918 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:27 smithi008 ceph-mon[67070]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T13:44:27.918 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:27 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2[78375]: 2024-04-14T13:44:27.739+0000 7f7ba381d7c0 -1 osd.2 97 log_to_monitors true 2024-04-14T13:44:27.918 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:44:27 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2[78375]: 2024-04-14T13:44:27.903+0000 7f7b94c75700 -1 osd.2 97 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:44:28.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:27 smithi012 ceph-mon[53289]: pgmap v143: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.0 MiB/s wr, 485 op/s; 148/1662 objects degraded (8.905%) 2024-04-14T13:44:28.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:27 smithi012 ceph-mon[53289]: from='osd.2 [v2:172.21.15.8:6818/2947889393,v1:172.21.15.8:6819/2947889393]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T13:44:28.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:27 smithi012 ceph-mon[53289]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T13:44:28.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 238784 10.81 MB/sec execute 422 sec latency 51.376 ms 2024-04-14T13:44:28.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:28 smithi008 ceph-mon[67070]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-14T13:44:28.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:28 smithi008 ceph-mon[67070]: osdmap e100: 8 total, 7 up, 8 in 2024-04-14T13:44:28.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:28 smithi008 ceph-mon[67070]: from='osd.2 [v2:172.21.15.8:6818/2947889393,v1:172.21.15.8:6819/2947889393]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:44:28.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:28 smithi008 ceph-mon[67070]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:44:29.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:28 smithi012 ceph-mon[53289]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-14T13:44:29.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:28 smithi012 ceph-mon[53289]: osdmap e100: 8 total, 7 up, 8 in 2024-04-14T13:44:29.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:28 smithi012 ceph-mon[53289]: from='osd.2 [v2:172.21.15.8:6818/2947889393,v1:172.21.15.8:6819/2947889393]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:44:29.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:28 smithi012 ceph-mon[53289]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:44:29.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 239014 10.80 MB/sec execute 423 sec latency 474.755 ms 2024-04-14T13:44:30.149 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:29 smithi008 ceph-mon[67070]: pgmap v145: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.4 MiB/s wr, 554 op/s; 148/1662 objects degraded (8.905%) 2024-04-14T13:44:30.149 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:29 smithi008 ceph-mon[67070]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T13:44:30.150 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:29 smithi008 ceph-mon[67070]: osd.2 [v2:172.21.15.8:6818/2947889393,v1:172.21.15.8:6819/2947889393] boot 2024-04-14T13:44:30.150 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:29 smithi008 ceph-mon[67070]: osdmap e101: 8 total, 8 up, 8 in 2024-04-14T13:44:30.150 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:44:30.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:29 smithi012 ceph-mon[53289]: pgmap v145: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.4 MiB/s wr, 554 op/s; 148/1662 objects degraded (8.905%) 2024-04-14T13:44:30.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:29 smithi012 ceph-mon[53289]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T13:44:30.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:29 smithi012 ceph-mon[53289]: osd.2 [v2:172.21.15.8:6818/2947889393,v1:172.21.15.8:6819/2947889393] boot 2024-04-14T13:44:30.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:29 smithi012 ceph-mon[53289]: osdmap e101: 8 total, 8 up, 8 in 2024-04-14T13:44:30.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:44:30.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 239164 10.78 MB/sec execute 424 sec latency 1474.784 ms 2024-04-14T13:44:30.925 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:30 smithi008 ceph-mon[67070]: osdmap e102: 8 total, 8 up, 8 in 2024-04-14T13:44:30.925 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:30 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:30.925 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:30 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:30.925 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:30 smithi008 ceph-mon[67070]: osdmap e103: 8 total, 8 up, 8 in 2024-04-14T13:44:31.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:30 smithi012 ceph-mon[53289]: osdmap e102: 8 total, 8 up, 8 in 2024-04-14T13:44:31.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:30 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:31.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:30 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:31.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:30 smithi012 ceph-mon[53289]: osdmap e103: 8 total, 8 up, 8 in 2024-04-14T13:44:31.355 INFO:teuthology.orchestra.run.smithi008.stdout: 5 239331 10.76 MB/sec execute 425 sec latency 2034.572 ms 2024-04-14T13:44:31.974 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:31 smithi008 ceph-mon[67070]: pgmap v148: 97 pgs: 16 peering, 3 active+undersized, 6 active+undersized+degraded, 72 active+clean; 353 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.3 MiB/s wr, 373 op/s; 50/1689 objects degraded (2.960%) 2024-04-14T13:44:31.974 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:31.974 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:32.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:31 smithi012 ceph-mon[53289]: pgmap v148: 97 pgs: 16 peering, 3 active+undersized, 6 active+undersized+degraded, 72 active+clean; 353 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.3 MiB/s wr, 373 op/s; 50/1689 objects degraded (2.960%) 2024-04-14T13:44:32.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:32.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:32.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 239620 10.76 MB/sec execute 426 sec latency 93.728 ms 2024-04-14T13:44:33.018 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:32 smithi008 ceph-mon[67070]: osdmap e104: 8 total, 8 up, 8 in 2024-04-14T13:44:33.018 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:32 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 50/1689 objects degraded (2.960%), 6 pgs degraded (PG_DEGRADED) 2024-04-14T13:44:33.018 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:32 smithi008 ceph-mon[67070]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T13:44:33.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:32 smithi012 ceph-mon[53289]: osdmap e104: 8 total, 8 up, 8 in 2024-04-14T13:44:33.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:32 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 50/1689 objects degraded (2.960%), 6 pgs degraded (PG_DEGRADED) 2024-04-14T13:44:33.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:32 smithi012 ceph-mon[53289]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T13:44:33.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 239858 10.75 MB/sec execute 427 sec latency 834.912 ms 2024-04-14T13:44:34.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:33 smithi008 ceph-mon[67070]: pgmap v152: 97 pgs: 16 peering, 1 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+undersized+degraded, 75 active+clean; 352 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 MiB/s wr, 641 op/s; 44/1668 objects degraded (2.638%); 42 KiB/s, 6 objects/s recovering 2024-04-14T13:44:34.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:33 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:34.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:33 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:44:34.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:33 smithi008 ceph-mon[67070]: osdmap e106: 8 total, 8 up, 8 in 2024-04-14T13:44:34.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:33 smithi012 ceph-mon[53289]: pgmap v152: 97 pgs: 16 peering, 1 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+undersized+degraded, 75 active+clean; 352 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 MiB/s wr, 641 op/s; 44/1668 objects degraded (2.638%); 42 KiB/s, 6 objects/s recovering 2024-04-14T13:44:34.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:33 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:34.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:33 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:44:34.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:33 smithi012 ceph-mon[53289]: osdmap e106: 8 total, 8 up, 8 in 2024-04-14T13:44:34.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 240098 10.73 MB/sec execute 428 sec latency 94.251 ms 2024-04-14T13:44:35.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 240356 10.71 MB/sec execute 429 sec latency 44.892 ms 2024-04-14T13:44:36.077 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:35 smithi008 ceph-mon[67070]: pgmap v154: 97 pgs: 2 peering, 1 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 3.1 MiB/s wr, 511 op/s; 21/1578 objects degraded (1.331%); 2.1 MiB/s, 29 objects/s recovering 2024-04-14T13:44:36.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:35 smithi012 ceph-mon[53289]: pgmap v154: 97 pgs: 2 peering, 1 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 3.1 MiB/s wr, 511 op/s; 21/1578 objects degraded (1.331%); 2.1 MiB/s, 29 objects/s recovering 2024-04-14T13:44:36.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 240596 10.71 MB/sec execute 430 sec latency 72.411 ms 2024-04-14T13:44:37.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 240816 10.70 MB/sec execute 431 sec latency 101.067 ms 2024-04-14T13:44:38.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:37 smithi008 ceph-mon[67070]: pgmap v155: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 2.6 MiB/s wr, 387 op/s; 1.5 MiB/s, 27 objects/s recovering 2024-04-14T13:44:38.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:37 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:38.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:37 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:38.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:37 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:44:38.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:37 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:44:38.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:37 smithi008 ceph-mon[67070]: pgmap v156: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 2.0 MiB/s wr, 237 op/s; 1.7 MiB/s, 26 objects/s recovering 2024-04-14T13:44:38.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:37 smithi008 ceph-mon[67070]: pgmap v157: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 2.2 MiB/s wr, 256 op/s; 1.8 MiB/s, 28 objects/s recovering 2024-04-14T13:44:38.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:37 smithi008 ceph-mon[67070]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/1578 objects degraded (1.331%), 1 pg degraded) 2024-04-14T13:44:38.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:37 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:38.128 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:37 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:38.128 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:37 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:44:38.128 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:37 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:44:38.128 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:37 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:44:38.128 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:37 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:44:38.128 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:37 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-14T13:44:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:37 smithi012 ceph-mon[53289]: pgmap v155: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 2.6 MiB/s wr, 387 op/s; 1.5 MiB/s, 27 objects/s recovering 2024-04-14T13:44:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:37 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:37 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:37 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:44:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:37 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:44:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:37 smithi012 ceph-mon[53289]: pgmap v156: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 2.0 MiB/s wr, 237 op/s; 1.7 MiB/s, 26 objects/s recovering 2024-04-14T13:44:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:37 smithi012 ceph-mon[53289]: pgmap v157: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 2.2 MiB/s wr, 256 op/s; 1.8 MiB/s, 28 objects/s recovering 2024-04-14T13:44:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:37 smithi012 ceph-mon[53289]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/1578 objects degraded (1.331%), 1 pg degraded) 2024-04-14T13:44:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:37 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:37 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:37 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:44:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:37 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:44:38.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:37 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:44:38.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:37 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:44:38.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:37 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-14T13:44:38.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 241158 10.71 MB/sec execute 432 sec latency 47.424 ms 2024-04-14T13:44:38.899 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:38 smithi008 ceph-mon[67070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-14T13:44:38.899 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:38 smithi008 ceph-mon[67070]: Upgrade: osd.3 is safe to restart 2024-04-14T13:44:39.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:38 smithi012 ceph-mon[53289]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-14T13:44:39.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:38 smithi012 ceph-mon[53289]: Upgrade: osd.3 is safe to restart 2024-04-14T13:44:39.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 241504 10.71 MB/sec execute 433 sec latency 62.708 ms 2024-04-14T13:44:39.925 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:39 smithi008 ceph-mon[67070]: pgmap v158: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 1.6 MiB/s wr, 190 op/s; 1.4 MiB/s, 21 objects/s recovering 2024-04-14T13:44:39.925 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:39 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:39.925 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:39 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T13:44:39.925 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:39 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:44:40.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:39 smithi012 ceph-mon[53289]: pgmap v158: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 1.6 MiB/s wr, 190 op/s; 1.4 MiB/s, 21 objects/s recovering 2024-04-14T13:44:40.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:39 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:40.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:39 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T13:44:40.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:39 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:44:40.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 241780 10.71 MB/sec execute 434 sec latency 67.093 ms 2024-04-14T13:44:40.929 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:40 smithi008 ceph-mon[67070]: Upgrade: Updating osd.3 2024-04-14T13:44:40.929 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:40 smithi008 ceph-mon[67070]: Deploying daemon osd.3 on smithi008 2024-04-14T13:44:41.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:40 smithi012 ceph-mon[53289]: Upgrade: Updating osd.3 2024-04-14T13:44:41.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:40 smithi012 ceph-mon[53289]: Deploying daemon osd.3 on smithi008 2024-04-14T13:44:41.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 242152 10.70 MB/sec execute 435 sec latency 30.743 ms 2024-04-14T13:44:41.941 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:41 smithi008 ceph-mon[67070]: pgmap v159: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.1 MiB/s wr, 328 op/s; 593 KiB/s, 17 objects/s recovering 2024-04-14T13:44:42.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:41 smithi012 ceph-mon[53289]: pgmap v159: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.1 MiB/s wr, 328 op/s; 593 KiB/s, 17 objects/s recovering 2024-04-14T13:44:42.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 242609 10.70 MB/sec execute 436 sec latency 26.218 ms 2024-04-14T13:44:43.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 242983 10.70 MB/sec execute 437 sec latency 65.591 ms 2024-04-14T13:44:43.991 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:43 smithi008 ceph-mon[67070]: pgmap v160: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.9 KiB/s rd, 3.6 MiB/s wr, 330 op/s; 51 KiB/s, 4 objects/s recovering 2024-04-14T13:44:43.991 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:43 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:44.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:43 smithi012 ceph-mon[53289]: pgmap v160: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.9 KiB/s rd, 3.6 MiB/s wr, 330 op/s; 51 KiB/s, 4 objects/s recovering 2024-04-14T13:44:44.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:43 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:44.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 243401 10.70 MB/sec execute 438 sec latency 44.477 ms 2024-04-14T13:44:45.251 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:45 smithi008 systemd[1]: Stopping Ceph osd.3 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:44:45.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 243746 10.68 MB/sec execute 439 sec latency 53.927 ms 2024-04-14T13:44:45.507 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:45 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3[45822]: 2024-04-14T13:44:45.480+0000 7fddc622a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:44:45.507 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:45 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3[45822]: 2024-04-14T13:44:45.480+0000 7fddc622a700 -1 osd.3 106 *** Got signal Terminated *** 2024-04-14T13:44:45.507 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:45 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3[45822]: 2024-04-14T13:44:45.480+0000 7fddc622a700 -1 osd.3 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T13:44:46.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:45 smithi008 ceph-mon[67070]: pgmap v161: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.7 MiB/s wr, 525 op/s 2024-04-14T13:44:46.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:45 smithi008 ceph-mon[67070]: osd.3 marked itself down and dead 2024-04-14T13:44:46.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:45 smithi012 ceph-mon[53289]: pgmap v161: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.7 MiB/s wr, 525 op/s 2024-04-14T13:44:46.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:45 smithi012 ceph-mon[53289]: osd.3 marked itself down and dead 2024-04-14T13:44:46.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 243928 10.66 MB/sec execute 440 sec latency 466.433 ms 2024-04-14T13:44:46.853 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:46 smithi008 podman[80297]: 2024-04-14 13:44:46.57867326 +0000 UTC m=+1.231156234 container died 66fca4b191be4fad29d894ccb46cf6dbe150eabd45169ff67253f3a21c645283 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, RELEASE=HEAD, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, release=754, version=8, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git) 2024-04-14T13:44:47.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:46 smithi008 ceph-mon[67070]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T13:44:47.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:46 smithi008 ceph-mon[67070]: osdmap e107: 8 total, 7 up, 8 in 2024-04-14T13:44:47.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:46 smithi008 ceph-mon[67070]: osdmap e108: 8 total, 7 up, 8 in 2024-04-14T13:44:47.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:46 smithi012 ceph-mon[53289]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T13:44:47.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:46 smithi012 ceph-mon[53289]: osdmap e107: 8 total, 7 up, 8 in 2024-04-14T13:44:47.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:46 smithi012 ceph-mon[53289]: osdmap e108: 8 total, 7 up, 8 in 2024-04-14T13:44:47.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 244090 10.65 MB/sec execute 441 sec latency 1059.158 ms 2024-04-14T13:44:47.949 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:47 smithi008 ceph-mon[67070]: pgmap v163: 97 pgs: 10 peering, 12 stale+active+clean, 75 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.8 MiB/s wr, 441 op/s 2024-04-14T13:44:47.949 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:47 smithi008 ceph-mon[67070]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-14T13:44:48.201 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:47 smithi008 podman[80297]: 2024-04-14 13:44:47.988758221 +0000 UTC m=+2.641241184 container cleanup 66fca4b191be4fad29d894ccb46cf6dbe150eabd45169ff67253f3a21c645283 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3, distribution-scope=public, version=8, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, vcs-type=git, name=centos-stream, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., release=754, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8) 2024-04-14T13:44:48.201 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:47 smithi008 bash[80297]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3 2024-04-14T13:44:48.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:47 smithi012 ceph-mon[53289]: pgmap v163: 97 pgs: 10 peering, 12 stale+active+clean, 75 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.8 MiB/s wr, 441 op/s 2024-04-14T13:44:48.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:47 smithi012 ceph-mon[53289]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-14T13:44:48.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 244487 10.66 MB/sec execute 442 sec latency 38.746 ms 2024-04-14T13:44:49.106 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:44:49.235 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:49 smithi008 podman[80315]: 2024-04-14 13:44:49.033352608 +0000 UTC m=+2.459500110 container remove 66fca4b191be4fad29d894ccb46cf6dbe150eabd45169ff67253f3a21c645283 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , architecture=x86_64, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, vendor=Red Hat, Inc., io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, RELEASE=HEAD, GIT_BRANCH=HEAD, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0) 2024-04-14T13:44:49.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:49.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:44:49.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:49 smithi008 ceph-mon[67070]: pgmap v165: 97 pgs: 1 active+undersized, 3 active+undersized+degraded, 10 peering, 11 stale+active+clean, 72 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.0 MiB/s wr, 544 op/s; 12/1779 objects degraded (0.675%) 2024-04-14T13:44:49.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 244884 10.67 MB/sec execute 443 sec latency 38.369 ms 2024-04-14T13:44:49.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:44:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:49 smithi012 ceph-mon[53289]: pgmap v165: 97 pgs: 1 active+undersized, 3 active+undersized+degraded, 10 peering, 11 stale+active+clean, 72 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.0 MiB/s wr, 544 op/s; 12/1779 objects degraded (0.675%) 2024-04-14T13:44:49.737 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:49 smithi008 podman[80442]: 2024-04-14 13:44:49.558991176 +0000 UTC m=+0.027137594 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:44:49.737 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:49 smithi008 podman[80442]: 2024-04-14 13:44:49.676402847 +0000 UTC m=+0.144549257 container create b525642623c5fe4a4a4b2c39bfbef79d7dd0f0ca4ec38702fbe20aa84bf0b24f (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:44:50.135 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:44:50.135 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (6m) 20s ago 15m 15.2M - 0.23.0 ba2b418f427c cfddb9a7f858 2024-04-14T13:44:50.135 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (2m) 20s ago 15m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 1e69eabdcc36 2024-04-14T13:44:50.135 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (2m) 2m ago 15m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 60c7a2992eb0 2024-04-14T13:44:50.135 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 20s ago 15m - - 2024-04-14T13:44:50.135 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (9m) 20s ago 9m 38.7M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:44:50.135 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (9m) 2m ago 9m 14.4M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:44:50.135 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (5m) 20s ago 17m 507M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:44:50.135 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (4m) 2m ago 15m 451M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:44:50.135 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (4m) 20s ago 17m 48.6M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 673dd6f4c8d8 2024-04-14T13:44:50.135 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (3m) 2m ago 14m 35.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 d498dec29c4f 2024-04-14T13:44:50.136 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (9m) 20s ago 9m 281M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:44:50.136 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (9m) 2m ago 9m 88.5M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:44:50.136 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (6m) 20s ago 15m 11.3M - 1.3.1 1dbe0e931976 646dfca70be4 2024-04-14T13:44:50.136 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (5m) 2m ago 14m 18.4M - 1.3.1 1dbe0e931976 0a60dcd4044c 2024-04-14T13:44:50.136 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (2m) 20s ago 14m 316M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 29997ccc66ac 2024-04-14T13:44:50.136 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (76s) 20s ago 13m 251M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 97df88f10bd6 2024-04-14T13:44:50.136 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (29s) 20s ago 13m 12.4M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 9325b527c16c 2024-04-14T13:44:50.136 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (12m) 20s ago 12m 431M 1937M 17.2.0 e1d6a67b021e 66fca4b191be 2024-04-14T13:44:50.136 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (12m) 2m ago 12m 614M 2705M 17.2.0 e1d6a67b021e 43b02cb64951 2024-04-14T13:44:50.136 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (12m) 2m ago 12m 430M 2705M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:44:50.136 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (11m) 2m ago 11m 341M 2705M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:44:50.136 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (11m) 2m ago 11m 552M 2705M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:44:50.136 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (4m) 20s ago 14m 69.8M - 2.33.4 514e6a882f6e 12f97f75d620 2024-04-14T13:44:50.243 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:50 smithi008 ceph-mon[67070]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:44:50.243 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:50 smithi008 ceph-mon[67070]: Health check failed: Degraded data redundancy: 12/1779 objects degraded (0.675%), 3 pgs degraded (PG_DEGRADED) 2024-04-14T13:44:50.243 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:50 smithi008 ceph-mon[67070]: from='client.44239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:44:50.243 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:49 smithi008 podman[80442]: 2024-04-14 13:44:49.988436428 +0000 UTC m=+0.456582845 container init b525642623c5fe4a4a4b2c39bfbef79d7dd0f0ca4ec38702fbe20aa84bf0b24f (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, ceph=True) 2024-04-14T13:44:50.243 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:49 smithi008 podman[80442]: 2024-04-14 13:44:49.995350408 +0000 UTC m=+0.463496826 container start b525642623c5fe4a4a4b2c39bfbef79d7dd0f0ca4ec38702fbe20aa84bf0b24f (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, ceph=True) 2024-04-14T13:44:50.243 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:50 smithi008 podman[80442]: 2024-04-14 13:44:50.157165567 +0000 UTC m=+0.625311986 container attach b525642623c5fe4a4a4b2c39bfbef79d7dd0f0ca4ec38702fbe20aa84bf0b24f (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-14T13:44:50.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 245185 10.67 MB/sec execute 444 sec latency 62.297 ms 2024-04-14T13:44:50.494 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:50 smithi008 podman[80442]: 2024-04-14 13:44:50.462391006 +0000 UTC m=+0.930537435 container died b525642623c5fe4a4a4b2c39bfbef79d7dd0f0ca4ec38702fbe20aa84bf0b24f (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403) 2024-04-14T13:44:50.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:50 smithi012 ceph-mon[53289]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:44:50.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:50 smithi012 ceph-mon[53289]: Health check failed: Degraded data redundancy: 12/1779 objects degraded (0.675%), 3 pgs degraded (PG_DEGRADED) 2024-04-14T13:44:50.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:50 smithi012 ceph-mon[53289]: from='client.44239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:44:50.748 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:44:50.749 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:44:50.749 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:44:50.749 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:44:50.749 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:44:50.749 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:44:50.749 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:44:50.749 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:44:50.749 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-14T13:44:50.749 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T13:44:50.749 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:44:50.749 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:44:50.749 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:44:50.749 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:44:50.749 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:44:50.750 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-14T13:44:50.750 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 7 2024-04-14T13:44:50.750 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:44:50.750 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:44:51.180 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:44:51.180 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:44:51.180 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:44:51.180 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:44:51.181 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-14T13:44:51.181 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-04-14T13:44:51.181 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-14T13:44:51.181 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-04-14T13:44:51.181 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:44:51.181 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "9/23 daemons upgraded", 2024-04-14T13:44:51.181 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T13:44:51.181 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:44:51.181 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:44:51.267 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:51 smithi008 ceph-mon[67070]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:44:51.267 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:51 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/575540486' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:44:51.267 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:51 smithi008 ceph-mon[67070]: pgmap v166: 97 pgs: 11 active+undersized, 20 active+undersized+degraded, 10 peering, 56 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.2 MiB/s wr, 488 op/s; 181/1722 objects degraded (10.511%) 2024-04-14T13:44:51.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 245449 10.67 MB/sec execute 445 sec latency 42.987 ms 2024-04-14T13:44:51.451 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:51 smithi012 ceph-mon[53289]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:44:51.451 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:51 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/575540486' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:44:51.451 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:51 smithi012 ceph-mon[53289]: pgmap v166: 97 pgs: 11 active+undersized, 20 active+undersized+degraded, 10 peering, 56 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.2 MiB/s wr, 488 op/s; 181/1722 objects degraded (10.511%) 2024-04-14T13:44:51.520 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:51 smithi008 podman[80513]: 2024-04-14 13:44:51.514220178 +0000 UTC m=+1.038691465 container remove b525642623c5fe4a4a4b2c39bfbef79d7dd0f0ca4ec38702fbe20aa84bf0b24f (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240403, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, RELEASE=main-a971fd5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-14T13:44:51.697 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Reduced data availability: 3 pgs inactive, 3 pgs peering; Degraded data redundancy: 181/1722 objects degraded (10.511%), 20 pgs degraded 2024-04-14T13:44:51.698 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:44:51.698 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-14T13:44:51.698 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-14T13:44:51.698 INFO:teuthology.orchestra.run.smithi008.stdout: osd.3 (root=default,host=smithi008) is down 2024-04-14T13:44:51.698 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs inactive, 3 pgs peering 2024-04-14T13:44:51.699 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.10 is stuck peering for 69s, current state peering, last acting [1,6] 2024-04-14T13:44:51.699 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.14 is stuck peering for 69s, current state peering, last acting [1,7] 2024-04-14T13:44:51.699 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.1d is stuck peering for 69s, current state peering, last acting [1,4] 2024-04-14T13:44:51.699 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 181/1722 objects degraded (10.511%), 20 pgs degraded 2024-04-14T13:44:51.699 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1 is active+undersized+degraded, acting [2,0] 2024-04-14T13:44:51.699 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.e is active+undersized+degraded, acting [2,4] 2024-04-14T13:44:51.699 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.12 is active+undersized+degraded, acting [5,7] 2024-04-14T13:44:51.700 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.13 is active+undersized+degraded, acting [0,4] 2024-04-14T13:44:51.700 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.14 is active+undersized+degraded, acting [6,5] 2024-04-14T13:44:51.700 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.19 is active+undersized+degraded, acting [6,0] 2024-04-14T13:44:51.700 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1b is active+undersized+degraded, acting [7,6] 2024-04-14T13:44:51.700 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1 is active+undersized+degraded, acting [0,4] 2024-04-14T13:44:51.700 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.2 is active+undersized+degraded, acting [5,6] 2024-04-14T13:44:51.700 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.5 is active+undersized+degraded, acting [5,2] 2024-04-14T13:44:51.701 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.7 is active+undersized+degraded, acting [7,0] 2024-04-14T13:44:51.701 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.b is active+undersized+degraded, acting [0,4] 2024-04-14T13:44:51.701 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.c is active+undersized+degraded, acting [5,6] 2024-04-14T13:44:51.701 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.12 is active+undersized+degraded, acting [0,7] 2024-04-14T13:44:51.701 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.15 is active+undersized+degraded, acting [7,4] 2024-04-14T13:44:51.701 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.17 is active+undersized+degraded, acting [0,5] 2024-04-14T13:44:51.701 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.18 is active+undersized+degraded, acting [0,1] 2024-04-14T13:44:51.702 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1e is active+undersized+degraded, acting [6,2] 2024-04-14T13:44:51.702 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.0 is active+undersized+degraded, acting [7,0] 2024-04-14T13:44:51.702 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.19 is active+undersized+degraded, acting [2,0] 2024-04-14T13:44:51.776 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:51 smithi008 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-14T13:44:51.776 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:51 smithi008 systemd[1]: Stopped Ceph osd.3 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:44:51.776 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:51 smithi008 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.3.service: Consumed 5min 7.361s CPU time. 2024-04-14T13:44:52.037 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:51 smithi008 systemd[1]: Starting Ceph osd.3 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:44:52.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:52 smithi008 ceph-mon[67070]: from='client.44249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:44:52.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:52 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/3711803227' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:44:52.160 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:52 smithi008 podman[80659]: 2024-04-14 13:44:51.998962087 +0000 UTC m=+0.020040001 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:44:52.293 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:52 smithi008 podman[80659]: 2024-04-14 13:44:52.158182728 +0000 UTC m=+0.179260677 container create ead1823185ddee81b1971db222623689f714def3cb717c255479335d57ef5484 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-activate, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:44:52.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 245683 10.65 MB/sec execute 446 sec latency 49.052 ms 2024-04-14T13:44:52.547 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:52 smithi008 podman[80659]: 2024-04-14 13:44:52.504570512 +0000 UTC m=+0.525648416 container init ead1823185ddee81b1971db222623689f714def3cb717c255479335d57ef5484 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-14T13:44:52.547 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:52 smithi008 podman[80659]: 2024-04-14 13:44:52.514466071 +0000 UTC m=+0.535543979 container start ead1823185ddee81b1971db222623689f714def3cb717c255479335d57ef5484 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, ceph=True, io.buildah.version=1.33.5) 2024-04-14T13:44:52.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:52 smithi012 ceph-mon[53289]: from='client.44249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:44:52.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:52 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/3711803227' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:44:52.816 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:52 smithi008 podman[80659]: 2024-04-14 13:44:52.637456861 +0000 UTC m=+0.658534767 container attach ead1823185ddee81b1971db222623689f714def3cb717c255479335d57ef5484 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-activate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-14T13:44:53.076 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:53 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-activate[80673]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-14T13:44:53.076 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:53 smithi008 bash[80659]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-14T13:44:53.076 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:53 smithi008 bash[80659]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-14T13:44:53.076 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:53 smithi008 bash[80659]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-14T13:44:53.076 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:53 smithi008 bash[80659]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-14T13:44:53.076 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:53 smithi008 bash[80659]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-14T13:44:53.076 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:53 smithi008 bash[80659]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-14T13:44:53.076 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:53 smithi008 bash[80659]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-14T13:44:53.077 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:53 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-activate[80673]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-14T13:44:53.077 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:53 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-activate[80673]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-14T13:44:53.077 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:53 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-activate[80673]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-14T13:44:53.077 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:53 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-activate[80673]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-14T13:44:53.077 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:53 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-activate[80673]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-14T13:44:53.077 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:53 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-activate[80673]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-14T13:44:53.077 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:53 smithi008 podman[80659]: 2024-04-14 13:44:53.058818629 +0000 UTC m=+1.079896542 container died ead1823185ddee81b1971db222623689f714def3cb717c255479335d57ef5484 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T13:44:53.338 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:53 smithi008 ceph-mon[67070]: pgmap v167: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.7 MiB/s wr, 390 op/s; 218/1731 objects degraded (12.594%) 2024-04-14T13:44:53.357 INFO:teuthology.orchestra.run.smithi008.stdout: 5 245904 10.63 MB/sec execute 447 sec latency 52.812 ms 2024-04-14T13:44:53.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:53 smithi012 ceph-mon[53289]: pgmap v167: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.7 MiB/s wr, 390 op/s; 218/1731 objects degraded (12.594%) 2024-04-14T13:44:54.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 246128 10.62 MB/sec execute 448 sec latency 53.849 ms 2024-04-14T13:44:54.372 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:54 smithi008 ceph-mon[67070]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-04-14T13:44:54.372 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:54 smithi008 podman[80809]: 2024-04-14 13:44:54.121199291 +0000 UTC m=+1.050248209 container remove ead1823185ddee81b1971db222623689f714def3cb717c255479335d57ef5484 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-activate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-a971fd5, ceph=True, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-14T13:44:54.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:54 smithi012 ceph-mon[53289]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-04-14T13:44:54.623 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:54 smithi008 podman[80871]: 2024-04-14 13:44:54.272759787 +0000 UTC m=+0.020215628 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:44:54.623 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:54 smithi008 podman[80871]: 2024-04-14 13:44:54.572785707 +0000 UTC m=+0.320241539 container create 6868b5076d616a154d9e059abb78bb82367a53545d2bf983737ef791aaf44597 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:44:55.146 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:54 smithi008 podman[80871]: 2024-04-14 13:44:54.928452026 +0000 UTC m=+0.675907858 container init 6868b5076d616a154d9e059abb78bb82367a53545d2bf983737ef791aaf44597 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-14T13:44:55.146 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:54 smithi008 podman[80871]: 2024-04-14 13:44:54.938580146 +0000 UTC m=+0.686035979 container start 6868b5076d616a154d9e059abb78bb82367a53545d2bf983737ef791aaf44597 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-14T13:44:55.146 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:54 smithi008 ceph-osd[80890]: -- 172.21.15.8:0/692218029 <== mon.0 v2:172.21.15.8:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bad13cf680 con 0x55bad1398c00 2024-04-14T13:44:55.146 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:55 smithi008 bash[80871]: 6868b5076d616a154d9e059abb78bb82367a53545d2bf983737ef791aaf44597 2024-04-14T13:44:55.146 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:55 smithi008 systemd[1]: Started Ceph osd.3 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:44:55.147 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:55 smithi008 ceph-mon[67070]: pgmap v168: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.1 MiB/s wr, 505 op/s; 220/1731 objects degraded (12.709%) 2024-04-14T13:44:55.147 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:55 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:55.147 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:55 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:55.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 246395 10.61 MB/sec execute 449 sec latency 56.293 ms 2024-04-14T13:44:55.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:55 smithi012 ceph-mon[53289]: pgmap v168: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.1 MiB/s wr, 505 op/s; 220/1731 objects degraded (12.709%) 2024-04-14T13:44:55.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:55 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:55.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:55 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:44:55.655 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:44:55 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3[80886]: 2024-04-14T13:44:55.518+0000 7f36a00867c0 -1 Falling back to public interface 2024-04-14T13:44:56.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:56 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 220/1731 objects degraded (12.709%), 26 pgs degraded (PG_DEGRADED) 2024-04-14T13:44:56.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:56 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:44:56.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 246728 10.61 MB/sec execute 450 sec latency 66.293 ms 2024-04-14T13:44:56.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:56 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 220/1731 objects degraded (12.709%), 26 pgs degraded (PG_DEGRADED) 2024-04-14T13:44:56.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:56 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:44:57.356 INFO:teuthology.orchestra.run.smithi008.stdout: 5 247022 10.60 MB/sec execute 451 sec latency 78.200 ms 2024-04-14T13:44:57.387 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:57 smithi008 ceph-mon[67070]: pgmap v169: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.5 MiB/s wr, 451 op/s; 218/1698 objects degraded (12.839%) 2024-04-14T13:44:57.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:57 smithi012 ceph-mon[53289]: pgmap v169: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.5 MiB/s wr, 451 op/s; 218/1698 objects degraded (12.839%) 2024-04-14T13:44:58.357 INFO:teuthology.orchestra.run.smithi008.stdout: 5 247374 10.59 MB/sec execute 452 sec latency 61.699 ms 2024-04-14T13:44:59.357 INFO:teuthology.orchestra.run.smithi008.stdout: 5 247690 10.58 MB/sec execute 453 sec latency 38.124 ms 2024-04-14T13:45:00.201 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:44:59 smithi008 ceph-mon[67070]: pgmap v170: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.8 MiB/s wr, 376 op/s; 218/1698 objects degraded (12.839%) 2024-04-14T13:45:00.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:44:59 smithi012 ceph-mon[53289]: pgmap v170: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.8 MiB/s wr, 376 op/s; 218/1698 objects degraded (12.839%) 2024-04-14T13:45:00.357 INFO:teuthology.orchestra.run.smithi008.stdout: 5 248056 10.59 MB/sec execute 454 sec latency 38.206 ms 2024-04-14T13:45:01.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:01 smithi008 ceph-mon[67070]: pgmap v171: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.9 MiB/s wr, 451 op/s; 200/1605 objects degraded (12.461%) 2024-04-14T13:45:01.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:01 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 200/1605 objects degraded (12.461%), 26 pgs degraded (PG_DEGRADED) 2024-04-14T13:45:01.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:01 smithi012 ceph-mon[53289]: pgmap v171: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.9 MiB/s wr, 451 op/s; 200/1605 objects degraded (12.461%) 2024-04-14T13:45:01.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:01 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 200/1605 objects degraded (12.461%), 26 pgs degraded (PG_DEGRADED) 2024-04-14T13:45:01.360 INFO:teuthology.orchestra.run.smithi008.stdout: 5 248450 10.60 MB/sec execute 455 sec latency 39.019 ms 2024-04-14T13:45:01.482 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:45:01 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3[80886]: 2024-04-14T13:45:01.348+0000 7f36a00867c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-04-14T13:45:02.365 INFO:teuthology.orchestra.run.smithi008.stdout: 5 248771 10.61 MB/sec execute 456 sec latency 56.189 ms 2024-04-14T13:45:03.066 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:45:02 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3[80886]: 2024-04-14T13:45:02.951+0000 7f36a00867c0 -1 osd.3 106 log_to_monitors true 2024-04-14T13:45:03.360 INFO:teuthology.orchestra.run.smithi008.stdout: 5 249060 10.60 MB/sec execute 457 sec latency 60.905 ms 2024-04-14T13:45:04.061 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:45:04 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3[80886]: 2024-04-14T13:45:04.004+0000 7f36914de700 -1 osd.3 106 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:45:04.062 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:03 smithi008 ceph-mon[67070]: pgmap v172: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.8 MiB/s wr, 402 op/s; 202/1617 objects degraded (12.492%) 2024-04-14T13:45:04.062 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:03 smithi008 ceph-mon[67070]: from='osd.3 [v2:172.21.15.8:6826/2835849246,v1:172.21.15.8:6827/2835849246]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T13:45:04.062 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:03 smithi008 ceph-mon[67070]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T13:45:04.062 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:04.062 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:45:04.062 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:04.062 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:04.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:03 smithi012 ceph-mon[53289]: pgmap v172: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.8 MiB/s wr, 402 op/s; 202/1617 objects degraded (12.492%) 2024-04-14T13:45:04.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:03 smithi012 ceph-mon[53289]: from='osd.3 [v2:172.21.15.8:6826/2835849246,v1:172.21.15.8:6827/2835849246]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T13:45:04.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:03 smithi012 ceph-mon[53289]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T13:45:04.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:04.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:45:04.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:04.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:04.360 INFO:teuthology.orchestra.run.smithi008.stdout: 5 249271 10.58 MB/sec execute 458 sec latency 39.990 ms 2024-04-14T13:45:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:04 smithi012 ceph-mon[53289]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-14T13:45:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:04 smithi012 ceph-mon[53289]: from='osd.3 [v2:172.21.15.8:6826/2835849246,v1:172.21.15.8:6827/2835849246]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:45:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:04 smithi012 ceph-mon[53289]: osdmap e109: 8 total, 7 up, 8 in 2024-04-14T13:45:05.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:04 smithi012 ceph-mon[53289]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:45:05.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:04 smithi008 ceph-mon[67070]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-14T13:45:05.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:04 smithi008 ceph-mon[67070]: from='osd.3 [v2:172.21.15.8:6826/2835849246,v1:172.21.15.8:6827/2835849246]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:45:05.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:04 smithi008 ceph-mon[67070]: osdmap e109: 8 total, 7 up, 8 in 2024-04-14T13:45:05.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:04 smithi008 ceph-mon[67070]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-14T13:45:05.360 INFO:teuthology.orchestra.run.smithi008.stdout: 5 249465 10.56 MB/sec execute 459 sec latency 404.457 ms 2024-04-14T13:45:06.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:05 smithi008 ceph-mon[67070]: pgmap v174: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 350 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.4 MiB/s wr, 427 op/s; 211/1695 objects degraded (12.448%) 2024-04-14T13:45:06.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:05 smithi008 ceph-mon[67070]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T13:45:06.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:05 smithi008 ceph-mon[67070]: osd.3 [v2:172.21.15.8:6826/2835849246,v1:172.21.15.8:6827/2835849246] boot 2024-04-14T13:45:06.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:05 smithi008 ceph-mon[67070]: osdmap e110: 8 total, 8 up, 8 in 2024-04-14T13:45:06.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:05 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:45:06.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:05 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:06.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:05 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:06.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:05 smithi012 ceph-mon[53289]: pgmap v174: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 350 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.4 MiB/s wr, 427 op/s; 211/1695 objects degraded (12.448%) 2024-04-14T13:45:06.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:05 smithi012 ceph-mon[53289]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T13:45:06.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:05 smithi012 ceph-mon[53289]: osd.3 [v2:172.21.15.8:6826/2835849246,v1:172.21.15.8:6827/2835849246] boot 2024-04-14T13:45:06.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:05 smithi012 ceph-mon[53289]: osdmap e110: 8 total, 8 up, 8 in 2024-04-14T13:45:06.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:05 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:45:06.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:05 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:06.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:05 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:06.360 INFO:teuthology.orchestra.run.smithi008.stdout: 5 249465 10.54 MB/sec execute 460 sec latency 1404.496 ms 2024-04-14T13:45:07.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:06 smithi012 ceph-mon[53289]: osdmap e111: 8 total, 8 up, 8 in 2024-04-14T13:45:07.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:06 smithi012 ceph-mon[53289]: pgmap v177: 97 pgs: 2 remapped+peering, 13 active+undersized, 21 active+undersized+degraded, 61 active+clean; 346 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.3 MiB/s wr, 341 op/s; 174/1692 objects degraded (10.284%); 0 B/s, 0 objects/s recovering 2024-04-14T13:45:07.342 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:06 smithi008 ceph-mon[67070]: osdmap e111: 8 total, 8 up, 8 in 2024-04-14T13:45:07.342 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:06 smithi008 ceph-mon[67070]: pgmap v177: 97 pgs: 2 remapped+peering, 13 active+undersized, 21 active+undersized+degraded, 61 active+clean; 346 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.3 MiB/s wr, 341 op/s; 174/1692 objects degraded (10.284%); 0 B/s, 0 objects/s recovering 2024-04-14T13:45:07.360 INFO:teuthology.orchestra.run.smithi008.stdout: 5 249466 10.52 MB/sec execute 461 sec latency 2403.813 ms 2024-04-14T13:45:08.148 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:07 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 174/1692 objects degraded (10.284%), 21 pgs degraded (PG_DEGRADED) 2024-04-14T13:45:08.148 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:07 smithi008 ceph-mon[67070]: osdmap e112: 8 total, 8 up, 8 in 2024-04-14T13:45:08.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:07 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 174/1692 objects degraded (10.284%), 21 pgs degraded (PG_DEGRADED) 2024-04-14T13:45:08.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:07 smithi012 ceph-mon[53289]: osdmap e112: 8 total, 8 up, 8 in 2024-04-14T13:45:08.360 INFO:teuthology.orchestra.run.smithi008.stdout: 5 249585 10.50 MB/sec execute 462 sec latency 2410.943 ms 2024-04-14T13:45:09.257 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:08 smithi008 ceph-mon[67070]: osdmap e113: 8 total, 8 up, 8 in 2024-04-14T13:45:09.257 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:08 smithi008 ceph-mon[67070]: pgmap v180: 97 pgs: 2 remapped+peering, 12 active+undersized, 2 active+undersized+remapped, 19 active+undersized+degraded, 62 active+clean; 350 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s wr, 195 op/s; 163/1695 objects degraded (9.617%); 2.2 MiB/s, 1 objects/s recovering 2024-04-14T13:45:09.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:08 smithi012 ceph-mon[53289]: osdmap e113: 8 total, 8 up, 8 in 2024-04-14T13:45:09.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:08 smithi012 ceph-mon[53289]: pgmap v180: 97 pgs: 2 remapped+peering, 12 active+undersized, 2 active+undersized+remapped, 19 active+undersized+degraded, 62 active+clean; 350 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s wr, 195 op/s; 163/1695 objects degraded (9.617%); 2.2 MiB/s, 1 objects/s recovering 2024-04-14T13:45:09.360 INFO:teuthology.orchestra.run.smithi008.stdout: 5 249666 10.48 MB/sec execute 463 sec latency 1038.131 ms 2024-04-14T13:45:10.044 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:10 smithi008 ceph-mon[67070]: osdmap e114: 8 total, 8 up, 8 in 2024-04-14T13:45:10.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:10 smithi012 ceph-mon[53289]: osdmap e114: 8 total, 8 up, 8 in 2024-04-14T13:45:10.360 INFO:teuthology.orchestra.run.smithi008.stdout: 5 249901 10.47 MB/sec execute 464 sec latency 994.560 ms 2024-04-14T13:45:11.071 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:11 smithi008 ceph-mon[67070]: osdmap e115: 8 total, 8 up, 8 in 2024-04-14T13:45:11.071 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:11 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:11.071 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:11 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:11.071 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:11 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:45:11.071 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:11 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:45:11.071 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:11 smithi008 ceph-mon[67070]: pgmap v183: 97 pgs: 4 peering, 2 remapped+peering, 2 active+undersized+remapped, 89 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 2.0 MiB/s wr, 271 op/s; 3.9 MiB/s, 48 objects/s recovering 2024-04-14T13:45:11.071 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:11 smithi008 ceph-mon[67070]: pgmap v184: 97 pgs: 4 peering, 2 remapped+peering, 2 active+undersized+remapped, 89 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 2.0 MiB/s wr, 278 op/s; 4.0 MiB/s, 50 objects/s recovering 2024-04-14T13:45:11.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:11 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:11.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:11 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:11.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:11 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:45:11.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:11 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:45:11.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:11 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:45:11.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:11 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:45:11.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:11 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T13:45:11.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:11 smithi008 ceph-mon[67070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T13:45:11.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:11 smithi008 ceph-mon[67070]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-04-14T13:45:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:11 smithi012 ceph-mon[53289]: osdmap e115: 8 total, 8 up, 8 in 2024-04-14T13:45:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:11 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:11 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:11 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:45:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:11 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:45:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:11 smithi012 ceph-mon[53289]: pgmap v183: 97 pgs: 4 peering, 2 remapped+peering, 2 active+undersized+remapped, 89 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 2.0 MiB/s wr, 271 op/s; 3.9 MiB/s, 48 objects/s recovering 2024-04-14T13:45:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:11 smithi012 ceph-mon[53289]: pgmap v184: 97 pgs: 4 peering, 2 remapped+peering, 2 active+undersized+remapped, 89 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 2.0 MiB/s wr, 278 op/s; 4.0 MiB/s, 50 objects/s recovering 2024-04-14T13:45:11.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:11 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:11.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:11 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:11.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:11 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:45:11.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:11 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:45:11.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:11 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:45:11.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:11 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:45:11.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:11 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T13:45:11.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:11 smithi012 ceph-mon[53289]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T13:45:11.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:11 smithi012 ceph-mon[53289]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-04-14T13:45:11.362 INFO:teuthology.orchestra.run.smithi008.stdout: 5 250189 10.46 MB/sec execute 465 sec latency 43.960 ms 2024-04-14T13:45:12.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:12 smithi008 ceph-mon[67070]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 163/1695 objects degraded (9.617%), 19 pgs degraded) 2024-04-14T13:45:12.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:12 smithi012 ceph-mon[53289]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 163/1695 objects degraded (9.617%), 19 pgs degraded) 2024-04-14T13:45:12.360 INFO:teuthology.orchestra.run.smithi008.stdout: 5 250457 10.46 MB/sec execute 466 sec latency 91.476 ms 2024-04-14T13:45:13.113 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:13 smithi008 ceph-mon[67070]: pgmap v185: 97 pgs: 4 peering, 1 remapped+peering, 2 active+undersized+remapped, 90 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 894 KiB/s wr, 63 op/s; 1.3 MiB/s, 43 objects/s recovering 2024-04-14T13:45:13.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:13 smithi012 ceph-mon[53289]: pgmap v185: 97 pgs: 4 peering, 1 remapped+peering, 2 active+undersized+remapped, 90 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 894 KiB/s wr, 63 op/s; 1.3 MiB/s, 43 objects/s recovering 2024-04-14T13:45:13.360 INFO:teuthology.orchestra.run.smithi008.stdout: 5 250835 10.45 MB/sec execute 467 sec latency 46.203 ms 2024-04-14T13:45:14.360 INFO:teuthology.orchestra.run.smithi008.stdout: 5 251272 10.45 MB/sec execute 468 sec latency 29.776 ms 2024-04-14T13:45:15.360 INFO:teuthology.orchestra.run.smithi008.stdout: 5 251628 10.46 MB/sec execute 469 sec latency 40.364 ms 2024-04-14T13:45:15.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:15 smithi008 ceph-mon[67070]: pgmap v186: 97 pgs: 2 peering, 95 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.3 MiB/s wr, 413 op/s; 1.2 MiB/s, 45 objects/s recovering 2024-04-14T13:45:16.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:15 smithi012 ceph-mon[53289]: pgmap v186: 97 pgs: 2 peering, 95 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.3 MiB/s wr, 413 op/s; 1.2 MiB/s, 45 objects/s recovering 2024-04-14T13:45:16.360 INFO:teuthology.orchestra.run.smithi008.stdout: 5 252097 10.48 MB/sec execute 470 sec latency 36.218 ms 2024-04-14T13:45:17.368 INFO:teuthology.orchestra.run.smithi008.stdout: 5 252438 10.47 MB/sec execute 471 sec latency 46.988 ms 2024-04-14T13:45:17.706 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:17 smithi008 ceph-mon[67070]: pgmap v187: 97 pgs: 97 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.9 MiB/s wr, 327 op/s; 884 KiB/s, 27 objects/s recovering 2024-04-14T13:45:18.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:17 smithi012 ceph-mon[53289]: pgmap v187: 97 pgs: 97 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.9 MiB/s wr, 327 op/s; 884 KiB/s, 27 objects/s recovering 2024-04-14T13:45:18.368 INFO:teuthology.orchestra.run.smithi008.stdout: 5 252728 10.47 MB/sec execute 472 sec latency 52.342 ms 2024-04-14T13:45:19.256 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:19 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:19.256 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:19 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:45:19.256 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:19 smithi008 ceph-mon[67070]: pgmap v188: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2 KiB/s rd, 2.6 MiB/s wr, 299 op/s; 164 KiB/s, 10 objects/s recovering 2024-04-14T13:45:19.369 INFO:teuthology.orchestra.run.smithi008.stdout: 5 253045 10.46 MB/sec execute 473 sec latency 37.708 ms 2024-04-14T13:45:19.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:19 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:19.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:19 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:45:19.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:19 smithi012 ceph-mon[53289]: pgmap v188: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2 KiB/s rd, 2.6 MiB/s wr, 299 op/s; 164 KiB/s, 10 objects/s recovering 2024-04-14T13:45:20.368 INFO:teuthology.orchestra.run.smithi008.stdout: 5 253332 10.45 MB/sec execute 474 sec latency 53.418 ms 2024-04-14T13:45:21.368 INFO:teuthology.orchestra.run.smithi008.stdout: 5 253605 10.44 MB/sec execute 475 sec latency 54.827 ms 2024-04-14T13:45:21.561 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:21 smithi008 ceph-mon[67070]: pgmap v189: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.7 MiB/s wr, 409 op/s; 139 KiB/s, 8 objects/s recovering 2024-04-14T13:45:22.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:21 smithi012 ceph-mon[53289]: pgmap v189: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.7 MiB/s wr, 409 op/s; 139 KiB/s, 8 objects/s recovering 2024-04-14T13:45:22.248 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:45:22.368 INFO:teuthology.orchestra.run.smithi008.stdout: 5 253891 10.44 MB/sec execute 476 sec latency 57.822 ms 2024-04-14T13:45:23.307 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:45:23.307 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (7m) 19s ago 16m 15.3M - 0.23.0 ba2b418f427c cfddb9a7f858 2024-04-14T13:45:23.307 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (3m) 19s ago 16m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 1e69eabdcc36 2024-04-14T13:45:23.307 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (3m) 3m ago 15m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 60c7a2992eb0 2024-04-14T13:45:23.307 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 19s ago 16m - - 2024-04-14T13:45:23.307 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (10m) 19s ago 10m 39.0M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:45:23.307 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (10m) 3m ago 10m 14.4M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:45:23.307 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (6m) 19s ago 17m 508M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:45:23.308 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (5m) 3m ago 15m 451M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:45:23.308 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (4m) 19s ago 17m 49.3M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 673dd6f4c8d8 2024-04-14T13:45:23.308 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (4m) 3m ago 15m 35.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 d498dec29c4f 2024-04-14T13:45:23.308 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (10m) 19s ago 10m 284M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:45:23.308 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (10m) 3m ago 10m 88.5M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:45:23.308 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (6m) 19s ago 15m 11.4M - 1.3.1 1dbe0e931976 646dfca70be4 2024-04-14T13:45:23.308 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (6m) 3m ago 15m 18.4M - 1.3.1 1dbe0e931976 0a60dcd4044c 2024-04-14T13:45:23.308 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (2m) 19s ago 14m 326M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 29997ccc66ac 2024-04-14T13:45:23.308 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (110s) 19s ago 14m 272M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 97df88f10bd6 2024-04-14T13:45:23.308 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (63s) 19s ago 13m 205M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 9325b527c16c 2024-04-14T13:45:23.308 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (29s) 19s ago 13m 14.7M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 6868b5076d61 2024-04-14T13:45:23.308 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (13m) 3m ago 13m 614M 2705M 17.2.0 e1d6a67b021e 43b02cb64951 2024-04-14T13:45:23.308 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (12m) 3m ago 12m 430M 2705M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:45:23.308 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (12m) 3m ago 12m 341M 2705M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:45:23.308 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (11m) 3m ago 11m 552M 2705M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:45:23.309 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (5m) 19s ago 15m 69.9M - 2.33.4 514e6a882f6e 12f97f75d620 2024-04-14T13:45:23.369 INFO:teuthology.orchestra.run.smithi008.stdout: 5 254152 10.42 MB/sec execute 477 sec latency 69.795 ms 2024-04-14T13:45:23.588 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:23 smithi008 ceph-mon[67070]: from='client.44257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:45:23.588 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:23 smithi008 ceph-mon[67070]: pgmap v190: 97 pgs: 97 active+clean; 345 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 4.2 MiB/s wr, 413 op/s; 116 KiB/s, 7 objects/s recovering 2024-04-14T13:45:23.588 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:23 smithi008 ceph-mon[67070]: from='client.44261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:45:23.588 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:23 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:23.871 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:45:23.871 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:45:23.871 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:45:23.872 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:45:23.872 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:45:23.872 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:45:23.872 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:45:23.872 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:45:23.872 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-14T13:45:23.872 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 4 2024-04-14T13:45:23.872 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:45:23.872 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:45:23.872 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:45:23.872 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:45:23.872 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:45:23.873 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-14T13:45:23.873 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 8 2024-04-14T13:45:23.873 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:45:23.873 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:45:24.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:23 smithi012 ceph-mon[53289]: from='client.44257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:45:24.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:23 smithi012 ceph-mon[53289]: pgmap v190: 97 pgs: 97 active+clean; 345 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 4.2 MiB/s wr, 413 op/s; 116 KiB/s, 7 objects/s recovering 2024-04-14T13:45:24.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:23 smithi012 ceph-mon[53289]: from='client.44261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:45:24.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:23 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:24.330 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:45:24.330 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:45:24.330 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:45:24.330 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:45:24.330 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-14T13:45:24.330 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-04-14T13:45:24.330 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-14T13:45:24.330 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-04-14T13:45:24.330 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:45:24.331 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "10/23 daemons upgraded", 2024-04-14T13:45:24.331 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T13:45:24.331 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:45:24.331 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:45:24.368 INFO:teuthology.orchestra.run.smithi008.stdout: 5 254528 10.42 MB/sec execute 478 sec latency 55.341 ms 2024-04-14T13:45:24.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:24 smithi008 ceph-mon[67070]: from='client.44265 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:45:24.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:24 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/3017514299' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:45:24.899 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T13:45:24.899 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:45:24.899 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-14T13:45:25.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:24 smithi012 ceph-mon[53289]: from='client.44265 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:45:25.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:24 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/3017514299' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:45:25.368 INFO:teuthology.orchestra.run.smithi008.stdout: 5 254827 10.41 MB/sec execute 479 sec latency 46.177 ms 2024-04-14T13:45:25.626 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:25 smithi008 ceph-mon[67070]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:45:25.626 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:25 smithi008 ceph-mon[67070]: pgmap v191: 97 pgs: 97 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.4 MiB/s wr, 584 op/s; 72 KiB/s, 4 objects/s recovering 2024-04-14T13:45:25.626 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:25 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/2707139297' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:45:26.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:25 smithi012 ceph-mon[53289]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:45:26.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:25 smithi012 ceph-mon[53289]: pgmap v191: 97 pgs: 97 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.4 MiB/s wr, 584 op/s; 72 KiB/s, 4 objects/s recovering 2024-04-14T13:45:26.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:25 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/2707139297' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:45:26.368 INFO:teuthology.orchestra.run.smithi008.stdout: 5 255152 10.42 MB/sec execute 480 sec latency 57.319 ms 2024-04-14T13:45:26.652 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:26 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T13:45:26.652 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:26 smithi012 ceph-mon[53289]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T13:45:26.652 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:26 smithi012 ceph-mon[53289]: Upgrade: osd.4 is safe to restart 2024-04-14T13:45:26.655 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:26 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T13:45:26.655 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:26 smithi008 ceph-mon[67070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T13:45:26.655 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:26 smithi008 ceph-mon[67070]: Upgrade: osd.4 is safe to restart 2024-04-14T13:45:27.368 INFO:teuthology.orchestra.run.smithi008.stdout: 5 255517 10.43 MB/sec execute 481 sec latency 75.152 ms 2024-04-14T13:45:27.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:27 smithi008 ceph-mon[67070]: pgmap v192: 97 pgs: 97 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.5 MiB/s wr, 427 op/s 2024-04-14T13:45:27.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:27 smithi012 ceph-mon[53289]: pgmap v192: 97 pgs: 97 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.5 MiB/s wr, 427 op/s 2024-04-14T13:45:28.368 INFO:teuthology.orchestra.run.smithi008.stdout: 5 255993 10.43 MB/sec execute 482 sec latency 30.946 ms 2024-04-14T13:45:28.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:28 smithi008 ceph-mon[67070]: Upgrade: Updating osd.4 2024-04-14T13:45:28.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:28 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:28.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:28 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T13:45:28.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:28 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:45:28.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:28 smithi008 ceph-mon[67070]: Deploying daemon osd.4 on smithi012 2024-04-14T13:45:29.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:28 smithi012 ceph-mon[53289]: Upgrade: Updating osd.4 2024-04-14T13:45:29.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:28 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:29.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:28 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T13:45:29.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:28 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:45:29.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:28 smithi012 ceph-mon[53289]: Deploying daemon osd.4 on smithi012 2024-04-14T13:45:29.368 INFO:teuthology.orchestra.run.smithi008.stdout: 5 256356 10.43 MB/sec execute 483 sec latency 38.585 ms 2024-04-14T13:45:29.936 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:29 smithi008 ceph-mon[67070]: pgmap v193: 97 pgs: 97 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.2 MiB/s wr, 418 op/s 2024-04-14T13:45:30.163 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:29 smithi012 ceph-mon[53289]: pgmap v193: 97 pgs: 97 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.2 MiB/s wr, 418 op/s 2024-04-14T13:45:30.368 INFO:teuthology.orchestra.run.smithi008.stdout: 5 256957 10.43 MB/sec execute 484 sec latency 58.984 ms 2024-04-14T13:45:31.369 INFO:teuthology.orchestra.run.smithi008.stdout: 5 257432 10.44 MB/sec execute 485 sec latency 46.589 ms 2024-04-14T13:45:31.990 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:31 smithi008 ceph-mon[67070]: pgmap v194: 97 pgs: 97 active+clean; 349 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 6.4 MiB/s wr, 615 op/s 2024-04-14T13:45:32.158 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:31 smithi012 ceph-mon[53289]: pgmap v194: 97 pgs: 97 active+clean; 349 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 6.4 MiB/s wr, 615 op/s 2024-04-14T13:45:32.159 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:31 smithi012 systemd[1]: Stopping Ceph osd.4 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:45:32.368 INFO:teuthology.orchestra.run.smithi008.stdout: 5 257695 10.43 MB/sec execute 486 sec latency 91.069 ms 2024-04-14T13:45:32.551 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:32 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[31414]: 2024-04-14T13:45:32.153+0000 7f891f963700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:45:32.551 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:32 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[31414]: 2024-04-14T13:45:32.153+0000 7f891f963700 -1 osd.4 115 *** Got signal Terminated *** 2024-04-14T13:45:32.551 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:32 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[31414]: 2024-04-14T13:45:32.153+0000 7f891f963700 -1 osd.4 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T13:45:33.042 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:32 smithi008 ceph-mon[67070]: osd.4 marked itself down and dead 2024-04-14T13:45:33.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:32 smithi012 ceph-mon[53289]: osd.4 marked itself down and dead 2024-04-14T13:45:33.369 INFO:teuthology.orchestra.run.smithi008.stdout: 5 257929 10.41 MB/sec execute 487 sec latency 483.944 ms 2024-04-14T13:45:33.800 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:33 smithi012 podman[57741]: 2024-04-14 13:45:33.518683516 +0000 UTC m=+1.497267593 container died 43b02cb649513a3ff98b53b6bb00f7c2fa3c45571a8d1ac895d76613e503b00c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4, maintainer=Guillaume Abrioux , io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, version=8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_CLEAN=True, distribution-scope=public, ceph=True) 2024-04-14T13:45:34.131 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:33 smithi008 ceph-mon[67070]: pgmap v195: 97 pgs: 97 active+clean; 350 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.6 MiB/s wr, 496 op/s 2024-04-14T13:45:34.131 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:33 smithi008 ceph-mon[67070]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T13:45:34.131 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:33 smithi008 ceph-mon[67070]: osdmap e116: 8 total, 7 up, 8 in 2024-04-14T13:45:34.131 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:33 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:34.131 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:33 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:45:34.182 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:33 smithi012 ceph-mon[53289]: pgmap v195: 97 pgs: 97 active+clean; 350 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.6 MiB/s wr, 496 op/s 2024-04-14T13:45:34.182 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:33 smithi012 ceph-mon[53289]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T13:45:34.182 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:33 smithi012 ceph-mon[53289]: osdmap e116: 8 total, 7 up, 8 in 2024-04-14T13:45:34.182 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:33 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:34.182 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:33 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:45:34.183 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:33 smithi012 podman[57741]: 2024-04-14 13:45:33.894514281 +0000 UTC m=+1.873098358 container cleanup 43b02cb649513a3ff98b53b6bb00f7c2fa3c45571a8d1ac895d76613e503b00c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., version=8, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, ceph=True, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True) 2024-04-14T13:45:34.183 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:33 smithi012 bash[57741]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4 2024-04-14T13:45:34.369 INFO:teuthology.orchestra.run.smithi008.stdout: 5 258145 10.40 MB/sec execute 488 sec latency 1029.300 ms 2024-04-14T13:45:34.545 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:34 smithi012 podman[57759]: 2024-04-14 13:45:34.179502406 +0000 UTC m=+0.655511107 container remove 43b02cb649513a3ff98b53b6bb00f7c2fa3c45571a8d1ac895d76613e503b00c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, architecture=x86_64, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD) 2024-04-14T13:45:34.801 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:34 smithi012 podman[57845]: 2024-04-14 13:45:34.447154703 +0000 UTC m=+0.021597402 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:45:34.801 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:34 smithi012 podman[57845]: 2024-04-14 13:45:34.572163849 +0000 UTC m=+0.146606532 container create 9fe4824ebe0b057789a810bed6871d2318fd12bd89a76304533684c9a141ff88 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-a971fd5) 2024-04-14T13:45:34.975 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:34 smithi008 ceph-mon[67070]: osdmap e117: 8 total, 7 up, 8 in 2024-04-14T13:45:35.203 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:34 smithi012 ceph-mon[53289]: osdmap e117: 8 total, 7 up, 8 in 2024-04-14T13:45:35.203 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:34 smithi012 podman[57845]: 2024-04-14 13:45:34.821822581 +0000 UTC m=+0.396265271 container init 9fe4824ebe0b057789a810bed6871d2318fd12bd89a76304533684c9a141ff88 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403) 2024-04-14T13:45:35.203 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:34 smithi012 podman[57845]: 2024-04-14 13:45:34.825951356 +0000 UTC m=+0.400394047 container start 9fe4824ebe0b057789a810bed6871d2318fd12bd89a76304533684c9a141ff88 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, GIT_CLEAN=True) 2024-04-14T13:45:35.203 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:34 smithi012 podman[57845]: 2024-04-14 13:45:34.925844998 +0000 UTC m=+0.500287702 container attach 9fe4824ebe0b057789a810bed6871d2318fd12bd89a76304533684c9a141ff88 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5) 2024-04-14T13:45:35.204 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:35 smithi012 podman[57845]: 2024-04-14 13:45:35.200233003 +0000 UTC m=+0.774675694 container died 9fe4824ebe0b057789a810bed6871d2318fd12bd89a76304533684c9a141ff88 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-04-14T13:45:35.369 INFO:teuthology.orchestra.run.smithi008.stdout: 5 258413 10.39 MB/sec execute 489 sec latency 45.325 ms 2024-04-14T13:45:35.966 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:35 smithi012 ceph-mon[53289]: pgmap v198: 97 pgs: 8 active+undersized, 16 active+undersized+degraded, 3 peering, 4 stale+active+clean, 66 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 5.4 MiB/s wr, 504 op/s; 146/1647 objects degraded (8.865%) 2024-04-14T13:45:35.966 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:35 smithi012 ceph-mon[53289]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-14T13:45:35.966 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:35 smithi012 ceph-mon[53289]: Health check failed: Degraded data redundancy: 146/1647 objects degraded (8.865%), 16 pgs degraded (PG_DEGRADED) 2024-04-14T13:45:35.982 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:35 smithi008 ceph-mon[67070]: pgmap v198: 97 pgs: 8 active+undersized, 16 active+undersized+degraded, 3 peering, 4 stale+active+clean, 66 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 5.4 MiB/s wr, 504 op/s; 146/1647 objects degraded (8.865%) 2024-04-14T13:45:35.982 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:35 smithi008 ceph-mon[67070]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-14T13:45:35.982 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:35 smithi008 ceph-mon[67070]: Health check failed: Degraded data redundancy: 146/1647 objects degraded (8.865%), 16 pgs degraded (PG_DEGRADED) 2024-04-14T13:45:36.225 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:35 smithi012 podman[57871]: 2024-04-14 13:45:35.963645899 +0000 UTC m=+0.754408967 container remove 9fe4824ebe0b057789a810bed6871d2318fd12bd89a76304533684c9a141ff88 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-deactivate, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:45:36.226 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:35 smithi012 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.4.service: Deactivated successfully. 2024-04-14T13:45:36.226 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:35 smithi012 systemd[1]: Stopped Ceph osd.4 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:45:36.226 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:35 smithi012 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.4.service: Consumed 5min 21.619s CPU time. 2024-04-14T13:45:36.369 INFO:teuthology.orchestra.run.smithi008.stdout: 5 258728 10.40 MB/sec execute 490 sec latency 39.963 ms 2024-04-14T13:45:36.484 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:36 smithi012 systemd[1]: Starting Ceph osd.4 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:45:36.484 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:36 smithi012 podman[57972]: 2024-04-14 13:45:36.362486641 +0000 UTC m=+0.022380966 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:45:36.795 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:36 smithi012 podman[57972]: 2024-04-14 13:45:36.481321122 +0000 UTC m=+0.141215452 container create 08a0f10b84b3b4d829ec27598ba6ed218143edca06a4655a596cf6393b92ab97 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T13:45:36.795 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:36 smithi012 podman[57972]: 2024-04-14 13:45:36.709675532 +0000 UTC m=+0.369569872 container init 08a0f10b84b3b4d829ec27598ba6ed218143edca06a4655a596cf6393b92ab97 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-activate, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=) 2024-04-14T13:45:36.795 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:36 smithi012 podman[57972]: 2024-04-14 13:45:36.713427683 +0000 UTC m=+0.373322012 container start 08a0f10b84b3b4d829ec27598ba6ed218143edca06a4655a596cf6393b92ab97 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-activate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-14T13:45:36.796 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:36 smithi012 podman[57972]: 2024-04-14 13:45:36.792721742 +0000 UTC m=+0.452616076 container attach 08a0f10b84b3b4d829ec27598ba6ed218143edca06a4655a596cf6393b92ab97 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-activate, RELEASE=main-a971fd5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS) 2024-04-14T13:45:37.369 INFO:teuthology.orchestra.run.smithi008.stdout: 5 258945 10.40 MB/sec execute 491 sec latency 71.242 ms 2024-04-14T13:45:37.551 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:37 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-activate[57985]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-14T13:45:37.551 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:37 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-activate[57985]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-14T13:45:37.551 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:37 smithi012 bash[57972]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-14T13:45:37.551 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:37 smithi012 bash[57972]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-14T13:45:37.551 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:37 smithi012 bash[57972]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-14T13:45:37.551 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:37 smithi012 bash[57972]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-14T13:45:37.552 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:37 smithi012 bash[57972]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-14T13:45:37.552 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:37 smithi012 bash[57972]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-14T13:45:37.552 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:37 smithi012 bash[57972]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-14T13:45:37.552 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:37 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-activate[57985]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-14T13:45:37.552 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:37 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-activate[57985]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-14T13:45:37.552 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:37 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-activate[57985]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-14T13:45:37.552 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:37 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-activate[57985]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-14T13:45:37.552 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:37 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-activate[57985]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-14T13:45:37.552 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:37 smithi012 podman[57972]: 2024-04-14 13:45:37.285011564 +0000 UTC m=+0.944905900 container died 08a0f10b84b3b4d829ec27598ba6ed218143edca06a4655a596cf6393b92ab97 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-activate, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-14T13:45:38.043 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:37 smithi008 ceph-mon[67070]: pgmap v199: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 3 peering, 2 stale+active+clean, 60 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.3 MiB/s wr, 652 op/s; 221/1608 objects degraded (13.744%) 2024-04-14T13:45:38.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:37 smithi012 ceph-mon[53289]: pgmap v199: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 3 peering, 2 stale+active+clean, 60 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.3 MiB/s wr, 652 op/s; 221/1608 objects degraded (13.744%) 2024-04-14T13:45:38.301 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:38 smithi012 podman[58112]: 2024-04-14 13:45:38.047646659 +0000 UTC m=+0.757162436 container remove 08a0f10b84b3b4d829ec27598ba6ed218143edca06a4655a596cf6393b92ab97 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-activate, RELEASE=main-a971fd5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-14T13:45:38.301 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:38 smithi012 podman[58168]: 2024-04-14 13:45:38.192980816 +0000 UTC m=+0.033187769 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:45:38.369 INFO:teuthology.orchestra.run.smithi008.stdout: 5 259270 10.39 MB/sec execute 492 sec latency 56.884 ms 2024-04-14T13:45:38.801 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:38 smithi012 podman[58168]: 2024-04-14 13:45:38.306955223 +0000 UTC m=+0.147162174 container create 178f9462f5ba00df66f38c91318e0b3f34c609ee840b9e4e72b5414e0ee75d7d (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-14T13:45:39.100 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:38 smithi008 ceph-mon[67070]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-14T13:45:39.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:38 smithi012 ceph-mon[53289]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-14T13:45:39.301 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:38 smithi012 podman[58168]: 2024-04-14 13:45:38.81828146 +0000 UTC m=+0.658488413 container init 178f9462f5ba00df66f38c91318e0b3f34c609ee840b9e4e72b5414e0ee75d7d (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-14T13:45:39.301 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:38 smithi012 podman[58168]: 2024-04-14 13:45:38.821544618 +0000 UTC m=+0.661751573 container start 178f9462f5ba00df66f38c91318e0b3f34c609ee840b9e4e72b5414e0ee75d7d (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:45:39.301 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:38 smithi012 ceph-osd[58185]: -- 172.21.15.12:0/25935878 <== mon.1 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5571ef239680 con 0x5571ef202c00 2024-04-14T13:45:39.301 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:38 smithi012 bash[58168]: 178f9462f5ba00df66f38c91318e0b3f34c609ee840b9e4e72b5414e0ee75d7d 2024-04-14T13:45:39.301 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:38 smithi012 systemd[1]: Started Ceph osd.4 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:45:39.369 INFO:teuthology.orchestra.run.smithi008.stdout: 5 259553 10.39 MB/sec execute 493 sec latency 55.290 ms 2024-04-14T13:45:39.800 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:39 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[58181]: 2024-04-14T13:45:39.367+0000 7f4a4fb9e7c0 -1 Falling back to public interface 2024-04-14T13:45:40.125 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:39 smithi008 ceph-mon[67070]: pgmap v200: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 2.9 MiB/s wr, 336 op/s; 276/1569 objects degraded (17.591%) 2024-04-14T13:45:40.125 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:39 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:40.125 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:39 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:40.125 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:39 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:45:40.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:39 smithi012 ceph-mon[53289]: pgmap v200: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 2.9 MiB/s wr, 336 op/s; 276/1569 objects degraded (17.591%) 2024-04-14T13:45:40.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:39 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:40.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:39 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:40.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:39 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:45:40.369 INFO:teuthology.orchestra.run.smithi008.stdout: 5 259743 10.38 MB/sec execute 494 sec latency 59.710 ms 2024-04-14T13:45:41.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:40 smithi012 ceph-mon[53289]: pgmap v201: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 5.5 MiB/s wr, 570 op/s; 287/1659 objects degraded (17.300%) 2024-04-14T13:45:41.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:40 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 287/1659 objects degraded (17.300%), 29 pgs degraded (PG_DEGRADED) 2024-04-14T13:45:41.144 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:40 smithi008 ceph-mon[67070]: pgmap v201: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 5.5 MiB/s wr, 570 op/s; 287/1659 objects degraded (17.300%) 2024-04-14T13:45:41.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:40 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 287/1659 objects degraded (17.300%), 29 pgs degraded (PG_DEGRADED) 2024-04-14T13:45:41.369 INFO:teuthology.orchestra.run.smithi008.stdout: 5 260020 10.38 MB/sec execute 495 sec latency 36.184 ms 2024-04-14T13:45:42.369 INFO:teuthology.orchestra.run.smithi008.stdout: 5 260305 10.36 MB/sec execute 496 sec latency 48.621 ms 2024-04-14T13:45:43.369 INFO:teuthology.orchestra.run.smithi008.stdout: 5 260603 10.36 MB/sec execute 497 sec latency 86.198 ms 2024-04-14T13:45:43.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:43 smithi008 ceph-mon[67070]: pgmap v202: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 4.8 MiB/s wr, 482 op/s; 288/1668 objects degraded (17.266%) 2024-04-14T13:45:43.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:43 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:43.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:43 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:44.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:43 smithi012 ceph-mon[53289]: pgmap v202: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 4.8 MiB/s wr, 482 op/s; 288/1668 objects degraded (17.266%) 2024-04-14T13:45:44.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:43 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:44.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:43 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:44.369 INFO:teuthology.orchestra.run.smithi008.stdout: 5 260849 10.35 MB/sec execute 498 sec latency 62.675 ms 2024-04-14T13:45:45.369 INFO:teuthology.orchestra.run.smithi008.stdout: 5 261113 10.34 MB/sec execute 499 sec latency 98.977 ms 2024-04-14T13:45:45.909 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:45 smithi012 ceph-mon[53289]: pgmap v203: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 349 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 5.7 MiB/s wr, 606 op/s; 296/1704 objects degraded (17.371%) 2024-04-14T13:45:45.910 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:45 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:45.910 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:45 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:46.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:45 smithi008 ceph-mon[67070]: pgmap v203: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 349 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 5.7 MiB/s wr, 606 op/s; 296/1704 objects degraded (17.371%) 2024-04-14T13:45:46.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:45 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:46.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:45 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:46.369 INFO:teuthology.orchestra.run.smithi008.stdout: 5 261407 10.33 MB/sec execute 500 sec latency 89.744 ms 2024-04-14T13:45:46.932 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:46 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 291/1683 objects degraded (17.291%), 29 pgs degraded (PG_DEGRADED) 2024-04-14T13:45:47.087 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:46 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 291/1683 objects degraded (17.291%), 29 pgs degraded (PG_DEGRADED) 2024-04-14T13:45:47.370 INFO:teuthology.orchestra.run.smithi008.stdout: 5 261697 10.32 MB/sec execute 501 sec latency 70.676 ms 2024-04-14T13:45:47.898 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:47 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[58181]: 2024-04-14T13:45:47.599+0000 7f4a4fb9e7c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-04-14T13:45:48.139 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:47 smithi008 ceph-mon[67070]: pgmap v204: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 348 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.5 MiB/s wr, 454 op/s; 291/1683 objects degraded (17.291%) 2024-04-14T13:45:48.258 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:47 smithi012 ceph-mon[53289]: pgmap v204: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 348 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.5 MiB/s wr, 454 op/s; 291/1683 objects degraded (17.291%) 2024-04-14T13:45:48.369 INFO:teuthology.orchestra.run.smithi008.stdout: 5 261893 10.31 MB/sec execute 502 sec latency 67.312 ms 2024-04-14T13:45:49.165 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:49.165 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:45:49.165 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:49 smithi008 ceph-mon[67070]: pgmap v205: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.9 MiB/s wr, 357 op/s; 289/1668 objects degraded (17.326%) 2024-04-14T13:45:49.165 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:49.165 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:49.165 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:45:49.165 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:45:49.165 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:49 smithi008 ceph-mon[67070]: pgmap v206: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.4 MiB/s wr, 411 op/s; 289/1668 objects degraded (17.326%) 2024-04-14T13:45:49.166 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:49 smithi008 ceph-mon[67070]: pgmap v207: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 2.7 MiB/s wr, 268 op/s; 289/1668 objects degraded (17.326%) 2024-04-14T13:45:49.166 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:49.166 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:49.166 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:45:49.166 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:45:49.166 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:45:49.166 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:45:49.166 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T13:45:49.166 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:49 smithi008 ceph-mon[67070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T13:45:49.166 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:49 smithi008 ceph-mon[67070]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-04-14T13:45:49.376 INFO:teuthology.orchestra.run.smithi008.stdout: 5 262339 10.33 MB/sec execute 503 sec latency 30.952 ms 2024-04-14T13:45:49.550 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:49 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[58181]: 2024-04-14T13:45:49.329+0000 7f4a4fb9e7c0 -1 osd.4 115 log_to_monitors true 2024-04-14T13:45:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:45:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:49 smithi012 ceph-mon[53289]: pgmap v205: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.9 MiB/s wr, 357 op/s; 289/1668 objects degraded (17.326%) 2024-04-14T13:45:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:45:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:45:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:49 smithi012 ceph-mon[53289]: pgmap v206: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.4 MiB/s wr, 411 op/s; 289/1668 objects degraded (17.326%) 2024-04-14T13:45:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:49 smithi012 ceph-mon[53289]: pgmap v207: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 2.7 MiB/s wr, 268 op/s; 289/1668 objects degraded (17.326%) 2024-04-14T13:45:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:49.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:45:49.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:45:49.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:45:49.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:45:49.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:45:49.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T13:45:49.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:49 smithi012 ceph-mon[53289]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T13:45:49.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:49 smithi012 ceph-mon[53289]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-04-14T13:45:50.178 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:50 smithi008 ceph-mon[67070]: from='osd.4 [v2:172.21.15.12:6800/2861750681,v1:172.21.15.12:6801/2861750681]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T13:45:50.178 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:50 smithi008 ceph-mon[67070]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T13:45:50.376 INFO:teuthology.orchestra.run.smithi008.stdout: 5 262801 10.33 MB/sec execute 504 sec latency 35.049 ms 2024-04-14T13:45:50.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:50 smithi012 ceph-mon[53289]: from='osd.4 [v2:172.21.15.12:6800/2861750681,v1:172.21.15.12:6801/2861750681]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T13:45:50.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:50 smithi012 ceph-mon[53289]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T13:45:50.551 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:45:50 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[58181]: 2024-04-14T13:45:50.126+0000 7f4a40ff6700 -1 osd.4 115 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:45:51.194 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:51 smithi008 ceph-mon[67070]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-14T13:45:51.194 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:51 smithi008 ceph-mon[67070]: from='osd.4 [v2:172.21.15.12:6800/2861750681,v1:172.21.15.12:6801/2861750681]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:45:51.194 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:51 smithi008 ceph-mon[67070]: osdmap e118: 8 total, 7 up, 8 in 2024-04-14T13:45:51.194 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:51 smithi008 ceph-mon[67070]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:45:51.194 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:51 smithi008 ceph-mon[67070]: pgmap v209: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 342 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.7 MiB/s wr, 364 op/s; 262/1590 objects degraded (16.478%) 2024-04-14T13:45:51.376 INFO:teuthology.orchestra.run.smithi008.stdout: 5 263177 10.34 MB/sec execute 505 sec latency 261.747 ms 2024-04-14T13:45:51.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:51 smithi012 ceph-mon[53289]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-14T13:45:51.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:51 smithi012 ceph-mon[53289]: from='osd.4 [v2:172.21.15.12:6800/2861750681,v1:172.21.15.12:6801/2861750681]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:45:51.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:51 smithi012 ceph-mon[53289]: osdmap e118: 8 total, 7 up, 8 in 2024-04-14T13:45:51.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:51 smithi012 ceph-mon[53289]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:45:51.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:51 smithi012 ceph-mon[53289]: pgmap v209: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 342 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.7 MiB/s wr, 364 op/s; 262/1590 objects degraded (16.478%) 2024-04-14T13:45:52.376 INFO:teuthology.orchestra.run.smithi008.stdout: 5 263277 10.32 MB/sec execute 506 sec latency 1261.780 ms 2024-04-14T13:45:52.378 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:52 smithi008 ceph-mon[67070]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T13:45:52.378 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:52 smithi008 ceph-mon[67070]: osd.4 [v2:172.21.15.12:6800/2861750681,v1:172.21.15.12:6801/2861750681] boot 2024-04-14T13:45:52.378 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:52 smithi008 ceph-mon[67070]: osdmap e119: 8 total, 8 up, 8 in 2024-04-14T13:45:52.379 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:52 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:45:52.451 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:52 smithi012 ceph-mon[53289]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T13:45:52.451 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:52 smithi012 ceph-mon[53289]: osd.4 [v2:172.21.15.12:6800/2861750681,v1:172.21.15.12:6801/2861750681] boot 2024-04-14T13:45:52.451 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:52 smithi012 ceph-mon[53289]: osdmap e119: 8 total, 8 up, 8 in 2024-04-14T13:45:52.451 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:52 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:45:53.376 INFO:teuthology.orchestra.run.smithi008.stdout: 5 263318 10.31 MB/sec execute 507 sec latency 2152.917 ms 2024-04-14T13:45:53.378 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:53 smithi008 ceph-mon[67070]: osdmap e120: 8 total, 8 up, 8 in 2024-04-14T13:45:53.378 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:53 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 262/1590 objects degraded (16.478%), 29 pgs degraded (PG_DEGRADED) 2024-04-14T13:45:53.379 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:53 smithi008 ceph-mon[67070]: pgmap v212: 97 pgs: 5 peering, 11 active+undersized, 26 active+undersized+degraded, 55 active+clean; 342 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 4.7 MiB/s wr, 528 op/s; 242/1605 objects degraded (15.078%) 2024-04-14T13:45:53.379 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:53 smithi008 ceph-mon[67070]: osdmap e121: 8 total, 8 up, 8 in 2024-04-14T13:45:53.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:53 smithi012 ceph-mon[53289]: osdmap e120: 8 total, 8 up, 8 in 2024-04-14T13:45:53.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:53 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 262/1590 objects degraded (16.478%), 29 pgs degraded (PG_DEGRADED) 2024-04-14T13:45:53.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:53 smithi012 ceph-mon[53289]: pgmap v212: 97 pgs: 5 peering, 11 active+undersized, 26 active+undersized+degraded, 55 active+clean; 342 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 4.7 MiB/s wr, 528 op/s; 242/1605 objects degraded (15.078%) 2024-04-14T13:45:53.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:53 smithi012 ceph-mon[53289]: osdmap e121: 8 total, 8 up, 8 in 2024-04-14T13:45:54.376 INFO:teuthology.orchestra.run.smithi008.stdout: 5 263598 10.30 MB/sec execute 508 sec latency 227.911 ms 2024-04-14T13:45:55.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:55 smithi008 ceph-mon[67070]: osdmap e122: 8 total, 8 up, 8 in 2024-04-14T13:45:55.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:55 smithi008 ceph-mon[67070]: pgmap v215: 97 pgs: 8 peering, 10 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+undersized+degraded, 67 active+clean; 340 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.5 MiB/s wr, 353 op/s; 135/1623 objects degraded (8.318%); 31 KiB/s, 35 objects/s recovering 2024-04-14T13:45:55.352 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:45:55.376 INFO:teuthology.orchestra.run.smithi008.stdout: 5 263779 10.29 MB/sec execute 509 sec latency 1014.600 ms 2024-04-14T13:45:55.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:55 smithi012 ceph-mon[53289]: osdmap e122: 8 total, 8 up, 8 in 2024-04-14T13:45:55.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:55 smithi012 ceph-mon[53289]: pgmap v215: 97 pgs: 8 peering, 10 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+undersized+degraded, 67 active+clean; 340 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.5 MiB/s wr, 353 op/s; 135/1623 objects degraded (8.318%); 31 KiB/s, 35 objects/s recovering 2024-04-14T13:45:56.197 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:56 smithi008 ceph-mon[67070]: osdmap e123: 8 total, 8 up, 8 in 2024-04-14T13:45:56.197 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:56 smithi008 ceph-mon[67070]: from='client.34360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:45:56.197 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:56 smithi008 ceph-mon[67070]: from='client.44281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:45:56.197 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:56 smithi008 ceph-mon[67070]: osdmap e124: 8 total, 8 up, 8 in 2024-04-14T13:45:56.376 INFO:teuthology.orchestra.run.smithi008.stdout: 5 264045 10.28 MB/sec execute 510 sec latency 237.322 ms 2024-04-14T13:45:56.383 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:45:56.383 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (7m) 53s ago 16m 15.3M - 0.23.0 ba2b418f427c cfddb9a7f858 2024-04-14T13:45:56.383 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (3m) 53s ago 16m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 1e69eabdcc36 2024-04-14T13:45:56.384 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (3m) 13s ago 16m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 60c7a2992eb0 2024-04-14T13:45:56.384 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 53s ago 16m - - 2024-04-14T13:45:56.384 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (11m) 53s ago 10m 39.0M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:45:56.384 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (10m) 13s ago 10m 15.0M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:45:56.384 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (6m) 53s ago 18m 508M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:45:56.384 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (5m) 13s ago 16m 452M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:45:56.384 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (5m) 53s ago 18m 49.3M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 673dd6f4c8d8 2024-04-14T13:45:56.384 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (4m) 13s ago 16m 40.1M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 d498dec29c4f 2024-04-14T13:45:56.384 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (10m) 53s ago 10m 284M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:45:56.384 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (10m) 13s ago 10m 90.9M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:45:56.384 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (7m) 53s ago 16m 11.4M - 1.3.1 1dbe0e931976 646dfca70be4 2024-04-14T13:45:56.384 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (6m) 13s ago 15m 20.7M - 1.3.1 1dbe0e931976 0a60dcd4044c 2024-04-14T13:45:56.384 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (3m) 53s ago 15m 326M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 29997ccc66ac 2024-04-14T13:45:56.384 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (2m) 53s ago 14m 272M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 97df88f10bd6 2024-04-14T13:45:56.384 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (96s) 53s ago 14m 205M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 9325b527c16c 2024-04-14T13:45:56.385 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (62s) 53s ago 14m 14.7M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 6868b5076d61 2024-04-14T13:45:56.385 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (18s) 13s ago 13m 15.1M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 178f9462f5ba 2024-04-14T13:45:56.385 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (13m) 13s ago 13m 610M 2705M 17.2.0 e1d6a67b021e bef2a23e2d78 2024-04-14T13:45:56.385 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (12m) 13s ago 12m 538M 2705M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:45:56.385 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (12m) 13s ago 12m 560M 2705M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:45:56.385 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (5m) 53s ago 15m 69.9M - 2.33.4 514e6a882f6e 12f97f75d620 2024-04-14T13:45:56.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:56 smithi012 ceph-mon[53289]: osdmap e123: 8 total, 8 up, 8 in 2024-04-14T13:45:56.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:56 smithi012 ceph-mon[53289]: from='client.34360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:45:56.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:56 smithi012 ceph-mon[53289]: from='client.44281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:45:56.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:56 smithi012 ceph-mon[53289]: osdmap e124: 8 total, 8 up, 8 in 2024-04-14T13:45:56.943 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:45:56.943 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:45:56.943 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:45:56.943 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:45:56.943 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:45:56.943 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:45:56.943 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:45:56.943 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:45:56.943 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-14T13:45:56.943 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 5 2024-04-14T13:45:56.943 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:45:56.944 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:45:56.944 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:45:56.944 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:45:56.944 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:45:56.944 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-14T13:45:56.944 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 9 2024-04-14T13:45:56.944 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:45:56.944 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:45:57.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:57 smithi008 ceph-mon[67070]: from='client.34368 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:45:57.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:57 smithi008 ceph-mon[67070]: pgmap v218: 97 pgs: 7 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.8 MiB/s wr, 613 op/s; 35/1590 objects degraded (2.201%); 671 KiB/s, 58 keys/s, 77 objects/s recovering 2024-04-14T13:45:57.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:57 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/310020894' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:45:57.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:57 smithi008 ceph-mon[67070]: osdmap e125: 8 total, 8 up, 8 in 2024-04-14T13:45:57.376 INFO:teuthology.orchestra.run.smithi008.stdout: 5 264284 10.28 MB/sec execute 511 sec latency 1032.139 ms 2024-04-14T13:45:57.436 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:45:57.436 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:45:57.436 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:45:57.436 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:45:57.436 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-14T13:45:57.436 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-04-14T13:45:57.437 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-14T13:45:57.437 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-04-14T13:45:57.437 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:45:57.437 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "11/23 daemons upgraded", 2024-04-14T13:45:57.437 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T13:45:57.437 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:45:57.437 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:45:57.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:57 smithi012 ceph-mon[53289]: from='client.34368 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:45:57.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:57 smithi012 ceph-mon[53289]: pgmap v218: 97 pgs: 7 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.8 MiB/s wr, 613 op/s; 35/1590 objects degraded (2.201%); 671 KiB/s, 58 keys/s, 77 objects/s recovering 2024-04-14T13:45:57.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:57 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/310020894' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:45:57.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:57 smithi012 ceph-mon[53289]: osdmap e125: 8 total, 8 up, 8 in 2024-04-14T13:45:58.052 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 35/1590 objects degraded (2.201%), 2 pgs degraded 2024-04-14T13:45:58.052 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:45:58.052 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-14T13:45:58.052 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 35/1590 objects degraded (2.201%), 2 pgs degraded 2024-04-14T13:45:58.052 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.11 is active+recovering+undersized+degraded+remapped, acting [7,6] 2024-04-14T13:45:58.052 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.13 is active+recovery_wait+undersized+degraded+remapped, acting [7,2] 2024-04-14T13:45:58.255 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:58 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 35/1590 objects degraded (2.201%), 2 pgs degraded (PG_DEGRADED) 2024-04-14T13:45:58.255 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:58 smithi008 ceph-mon[67070]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:45:58.255 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:58 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/3941229548' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:45:58.376 INFO:teuthology.orchestra.run.smithi008.stdout: 5 264689 10.27 MB/sec execute 512 sec latency 54.704 ms 2024-04-14T13:45:58.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:58 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 35/1590 objects degraded (2.201%), 2 pgs degraded (PG_DEGRADED) 2024-04-14T13:45:58.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:58 smithi012 ceph-mon[53289]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:45:58.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:58 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/3941229548' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:45:59.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:45:59 smithi008 ceph-mon[67070]: pgmap v220: 97 pgs: 7 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.4 MiB/s wr, 315 op/s; 35/1590 objects degraded (2.201%); 566 KiB/s, 51 keys/s, 59 objects/s recovering 2024-04-14T13:45:59.376 INFO:teuthology.orchestra.run.smithi008.stdout: 5 265049 10.26 MB/sec execute 513 sec latency 51.555 ms 2024-04-14T13:45:59.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:45:59 smithi012 ceph-mon[53289]: pgmap v220: 97 pgs: 7 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.4 MiB/s wr, 315 op/s; 35/1590 objects degraded (2.201%); 566 KiB/s, 51 keys/s, 59 objects/s recovering 2024-04-14T13:46:00.376 INFO:teuthology.orchestra.run.smithi008.stdout: 5 265409 10.26 MB/sec execute 514 sec latency 43.896 ms 2024-04-14T13:46:00.837 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:00 smithi008 ceph-mon[67070]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/1590 objects degraded (2.201%), 2 pgs degraded) 2024-04-14T13:46:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:00 smithi012 ceph-mon[53289]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/1590 objects degraded (2.201%), 2 pgs degraded) 2024-04-14T13:46:01.376 INFO:teuthology.orchestra.run.smithi008.stdout: 5 265773 10.26 MB/sec execute 515 sec latency 53.589 ms 2024-04-14T13:46:01.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:01 smithi008 ceph-mon[67070]: pgmap v221: 97 pgs: 97 active+clean; 335 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.4 MiB/s wr, 476 op/s; 427 KiB/s, 38 keys/s, 34 objects/s recovering 2024-04-14T13:46:02.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:01 smithi012 ceph-mon[53289]: pgmap v221: 97 pgs: 97 active+clean; 335 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.4 MiB/s wr, 476 op/s; 427 KiB/s, 38 keys/s, 34 objects/s recovering 2024-04-14T13:46:02.376 INFO:teuthology.orchestra.run.smithi008.stdout: 5 266172 10.26 MB/sec execute 516 sec latency 46.748 ms 2024-04-14T13:46:03.376 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:03 smithi008 ceph-mon[67070]: pgmap v222: 97 pgs: 97 active+clean; 337 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.4 MiB/s wr, 443 op/s; 20 KiB/s, 31 keys/s, 8 objects/s recovering 2024-04-14T13:46:03.376 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:03.376 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:46:03.376 INFO:teuthology.orchestra.run.smithi008.stdout: 5 266529 10.27 MB/sec execute 517 sec latency 35.188 ms 2024-04-14T13:46:03.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:03 smithi012 ceph-mon[53289]: pgmap v222: 97 pgs: 97 active+clean; 337 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.4 MiB/s wr, 443 op/s; 20 KiB/s, 31 keys/s, 8 objects/s recovering 2024-04-14T13:46:03.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:03.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:46:04.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 266838 10.28 MB/sec execute 518 sec latency 39.381 ms 2024-04-14T13:46:04.425 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:04 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T13:46:04.425 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:04 smithi008 ceph-mon[67070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T13:46:04.425 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:04 smithi008 ceph-mon[67070]: Upgrade: osd.5 is safe to restart 2024-04-14T13:46:04.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:04 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T13:46:04.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:04 smithi012 ceph-mon[53289]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T13:46:04.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:04 smithi012 ceph-mon[53289]: Upgrade: osd.5 is safe to restart 2024-04-14T13:46:05.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 267103 10.27 MB/sec execute 519 sec latency 58.384 ms 2024-04-14T13:46:05.452 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:05 smithi008 ceph-mon[67070]: pgmap v223: 97 pgs: 97 active+clean; 337 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.8 MiB/s wr, 504 op/s; 18 KiB/s, 27 keys/s, 7 objects/s recovering 2024-04-14T13:46:05.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:05 smithi012 ceph-mon[53289]: pgmap v223: 97 pgs: 97 active+clean; 337 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.8 MiB/s wr, 504 op/s; 18 KiB/s, 27 keys/s, 7 objects/s recovering 2024-04-14T13:46:06.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 267373 10.26 MB/sec execute 520 sec latency 69.419 ms 2024-04-14T13:46:07.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:07 smithi008 ceph-mon[67070]: Upgrade: Updating osd.5 2024-04-14T13:46:07.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:07 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:07.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:07 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T13:46:07.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:07 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:46:07.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:07 smithi008 ceph-mon[67070]: Deploying daemon osd.5 on smithi012 2024-04-14T13:46:07.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:07 smithi008 ceph-mon[67070]: pgmap v224: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.8 MiB/s wr, 422 op/s; 0 B/s, 4 objects/s recovering 2024-04-14T13:46:07.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 267618 10.25 MB/sec execute 521 sec latency 50.208 ms 2024-04-14T13:46:07.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:07 smithi012 ceph-mon[53289]: Upgrade: Updating osd.5 2024-04-14T13:46:07.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:07 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:07.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:07 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T13:46:07.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:07 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:46:07.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:07 smithi012 ceph-mon[53289]: Deploying daemon osd.5 on smithi012 2024-04-14T13:46:07.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:07 smithi012 ceph-mon[53289]: pgmap v224: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.8 MiB/s wr, 422 op/s; 0 B/s, 4 objects/s recovering 2024-04-14T13:46:08.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 267932 10.24 MB/sec execute 522 sec latency 45.995 ms 2024-04-14T13:46:09.267 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:09 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:09.268 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:09 smithi008 ceph-mon[67070]: pgmap v225: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.3 MiB/s wr, 409 op/s; 0 B/s, 3 objects/s recovering 2024-04-14T13:46:09.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 268317 10.24 MB/sec execute 523 sec latency 36.402 ms 2024-04-14T13:46:09.525 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:09 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:09.525 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:09 smithi012 ceph-mon[53289]: pgmap v225: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.3 MiB/s wr, 409 op/s; 0 B/s, 3 objects/s recovering 2024-04-14T13:46:09.801 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:09 smithi012 systemd[1]: Stopping Ceph osd.5 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:46:10.293 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:10 smithi008 ceph-mon[67070]: osd.5 marked itself down and dead 2024-04-14T13:46:10.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:10 smithi012 ceph-mon[53289]: osd.5 marked itself down and dead 2024-04-14T13:46:10.301 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:10 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[34450]: 2024-04-14T13:46:10.022+0000 7f8d3c9a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:46:10.301 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:10 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[34450]: 2024-04-14T13:46:10.022+0000 7f8d3c9a0700 -1 osd.5 125 *** Got signal Terminated *** 2024-04-14T13:46:10.301 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:10 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[34450]: 2024-04-14T13:46:10.022+0000 7f8d3c9a0700 -1 osd.5 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T13:46:10.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 268614 10.23 MB/sec execute 524 sec latency 144.120 ms 2024-04-14T13:46:11.139 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:10 smithi012 podman[59988]: 2024-04-14 13:46:10.872391945 +0000 UTC m=+0.979549788 container died bef2a23e2d7842105de9b8ce87033492037110eab9b00db538129a73fb7047cd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., distribution-scope=public, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, ceph=True, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, io.openshift.expose-services=, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, version=8) 2024-04-14T13:46:11.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 268654 10.21 MB/sec execute 525 sec latency 1015.001 ms 2024-04-14T13:46:11.468 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:11 smithi008 ceph-mon[67070]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T13:46:11.469 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:11 smithi008 ceph-mon[67070]: osdmap e126: 8 total, 7 up, 8 in 2024-04-14T13:46:11.469 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:11 smithi008 ceph-mon[67070]: pgmap v227: 97 pgs: 12 stale+active+clean, 85 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.7 MiB/s wr, 442 op/s 2024-04-14T13:46:11.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:11 smithi012 ceph-mon[53289]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T13:46:11.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:11 smithi012 ceph-mon[53289]: osdmap e126: 8 total, 7 up, 8 in 2024-04-14T13:46:11.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:11 smithi012 ceph-mon[53289]: pgmap v227: 97 pgs: 12 stale+active+clean, 85 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.7 MiB/s wr, 442 op/s 2024-04-14T13:46:11.940 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:11 smithi012 podman[59988]: 2024-04-14 13:46:11.677714782 +0000 UTC m=+1.784872622 container cleanup bef2a23e2d7842105de9b8ce87033492037110eab9b00db538129a73fb7047cd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_CLEAN=True, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, maintainer=Guillaume Abrioux , vcs-type=git, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, ceph=True, io.buildah.version=1.19.8) 2024-04-14T13:46:11.941 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:11 smithi012 bash[59988]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5 2024-04-14T13:46:12.219 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:11 smithi012 podman[60014]: 2024-04-14 13:46:11.937753508 +0000 UTC m=+1.062284443 container remove bef2a23e2d7842105de9b8ce87033492037110eab9b00db538129a73fb7047cd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, version=8, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, maintainer=Guillaume Abrioux , vcs-type=git, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., io.buildah.version=1.19.8, architecture=x86_64, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True) 2024-04-14T13:46:12.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:12 smithi008 ceph-mon[67070]: osdmap e127: 8 total, 7 up, 8 in 2024-04-14T13:46:12.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 269066 10.22 MB/sec execute 526 sec latency 45.563 ms 2024-04-14T13:46:12.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:12 smithi012 ceph-mon[53289]: osdmap e127: 8 total, 7 up, 8 in 2024-04-14T13:46:12.551 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:12 smithi012 podman[60099]: 2024-04-14 13:46:12.186712271 +0000 UTC m=+0.021750687 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:46:12.551 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:12 smithi012 podman[60099]: 2024-04-14 13:46:12.44720354 +0000 UTC m=+0.282241954 container create b3fe8b89b04396dbd24aafad4145a0c9a0a40702b3ba52190316955ea18f19cd (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-14T13:46:13.051 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:12 smithi012 podman[60099]: 2024-04-14 13:46:12.700049833 +0000 UTC m=+0.535088248 container init b3fe8b89b04396dbd24aafad4145a0c9a0a40702b3ba52190316955ea18f19cd (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, RELEASE=main-a971fd5, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-14T13:46:13.051 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:12 smithi012 podman[60099]: 2024-04-14 13:46:12.703644746 +0000 UTC m=+0.538683167 container start b3fe8b89b04396dbd24aafad4145a0c9a0a40702b3ba52190316955ea18f19cd (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, ceph=True, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=) 2024-04-14T13:46:13.051 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:12 smithi012 podman[60099]: 2024-04-14 13:46:12.800252444 +0000 UTC m=+0.635290875 container attach b3fe8b89b04396dbd24aafad4145a0c9a0a40702b3ba52190316955ea18f19cd (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=) 2024-04-14T13:46:13.280 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:13 smithi008 ceph-mon[67070]: pgmap v229: 97 pgs: 2 active+undersized, 3 active+undersized+degraded, 11 stale+active+clean, 81 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.6 MiB/s wr, 357 op/s; 13/1617 objects degraded (0.804%) 2024-04-14T13:46:13.365 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:13 smithi012 ceph-mon[53289]: pgmap v229: 97 pgs: 2 active+undersized, 3 active+undersized+degraded, 11 stale+active+clean, 81 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.6 MiB/s wr, 357 op/s; 13/1617 objects degraded (0.804%) 2024-04-14T13:46:13.365 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:13 smithi012 podman[60099]: 2024-04-14 13:46:13.061233461 +0000 UTC m=+0.896271877 container died b3fe8b89b04396dbd24aafad4145a0c9a0a40702b3ba52190316955ea18f19cd (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-14T13:46:13.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 269416 10.21 MB/sec execute 527 sec latency 65.020 ms 2024-04-14T13:46:14.051 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:13 smithi012 podman[60125]: 2024-04-14 13:46:13.66348219 +0000 UTC m=+0.596168952 container remove b3fe8b89b04396dbd24aafad4145a0c9a0a40702b3ba52190316955ea18f19cd (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-14T13:46:14.051 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:13 smithi012 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.5.service: Deactivated successfully. 2024-04-14T13:46:14.051 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:13 smithi012 systemd[1]: Stopped Ceph osd.5 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:46:14.051 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:13 smithi012 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.5.service: Consumed 5min 41.536s CPU time. 2024-04-14T13:46:14.051 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:13 smithi012 systemd[1]: Starting Ceph osd.5 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:46:14.377 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:14 smithi008 ceph-mon[67070]: Health check failed: Degraded data redundancy: 13/1617 objects degraded (0.804%), 3 pgs degraded (PG_DEGRADED) 2024-04-14T13:46:14.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 269781 10.22 MB/sec execute 528 sec latency 35.333 ms 2024-04-14T13:46:14.445 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:14 smithi012 ceph-mon[53289]: Health check failed: Degraded data redundancy: 13/1617 objects degraded (0.804%), 3 pgs degraded (PG_DEGRADED) 2024-04-14T13:46:14.445 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:14 smithi012 podman[60221]: 2024-04-14 13:46:14.04524786 +0000 UTC m=+0.021230326 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:46:14.445 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:14 smithi012 podman[60221]: 2024-04-14 13:46:14.181156044 +0000 UTC m=+0.157138509 container create e1a0405ab8eb31213202c7f79d88e1905269b35a5c7439a7b734f894eee6d1bd (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, ceph=True) 2024-04-14T13:46:14.445 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:14 smithi012 podman[60221]: 2024-04-14 13:46:14.442393504 +0000 UTC m=+0.418375969 container init e1a0405ab8eb31213202c7f79d88e1905269b35a5c7439a7b734f894eee6d1bd (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=main-a971fd5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-14T13:46:14.801 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:14 smithi012 podman[60221]: 2024-04-14 13:46:14.446964677 +0000 UTC m=+0.422947145 container start e1a0405ab8eb31213202c7f79d88e1905269b35a5c7439a7b734f894eee6d1bd (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-activate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, ceph=True) 2024-04-14T13:46:14.801 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:14 smithi012 podman[60221]: 2024-04-14 13:46:14.542456636 +0000 UTC m=+0.518439111 container attach e1a0405ab8eb31213202c7f79d88e1905269b35a5c7439a7b734f894eee6d1bd (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-activate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240403) 2024-04-14T13:46:15.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:15 smithi012 ceph-mon[53289]: pgmap v230: 97 pgs: 6 active+undersized, 17 active+undersized+degraded, 3 stale+active+clean, 71 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 3.9 MiB/s wr, 499 op/s; 121/1632 objects degraded (7.414%) 2024-04-14T13:46:15.301 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:14 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-activate[60235]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-14T13:46:15.301 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:14 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-activate[60235]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-14T13:46:15.301 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:14 smithi012 bash[60221]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-14T13:46:15.301 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:14 smithi012 bash[60221]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-14T13:46:15.301 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:14 smithi012 bash[60221]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-14T13:46:15.301 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:14 smithi012 bash[60221]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-14T13:46:15.301 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:14 smithi012 bash[60221]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-14T13:46:15.301 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:14 smithi012 bash[60221]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-14T13:46:15.302 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:14 smithi012 bash[60221]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-14T13:46:15.302 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:14 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-activate[60235]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-14T13:46:15.302 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:14 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-activate[60235]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-14T13:46:15.302 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:14 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-activate[60235]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-14T13:46:15.302 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:14 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-activate[60235]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-14T13:46:15.302 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:14 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-activate[60235]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-14T13:46:15.302 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:15 smithi012 podman[60221]: 2024-04-14 13:46:15.013995999 +0000 UTC m=+0.989978466 container died e1a0405ab8eb31213202c7f79d88e1905269b35a5c7439a7b734f894eee6d1bd (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-activate, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T13:46:15.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 270088 10.22 MB/sec execute 529 sec latency 50.502 ms 2024-04-14T13:46:15.427 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:15 smithi008 ceph-mon[67070]: pgmap v230: 97 pgs: 6 active+undersized, 17 active+undersized+degraded, 3 stale+active+clean, 71 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 3.9 MiB/s wr, 499 op/s; 121/1632 objects degraded (7.414%) 2024-04-14T13:46:16.110 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:15 smithi012 podman[60362]: 2024-04-14 13:46:15.856236151 +0000 UTC m=+0.835414267 container remove e1a0405ab8eb31213202c7f79d88e1905269b35a5c7439a7b734f894eee6d1bd (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=main-a971fd5, org.label-schema.build-date=20240403) 2024-04-14T13:46:16.110 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:16 smithi012 podman[60421]: 2024-04-14 13:46:15.98987494 +0000 UTC m=+0.024147402 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:46:16.371 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:16 smithi012 podman[60421]: 2024-04-14 13:46:16.107380563 +0000 UTC m=+0.141652999 container create 25e416621d4036239dc419587544f348c677403acfb703c339ba78306ceb4e3e (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-14T13:46:16.372 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:16 smithi012 podman[60421]: 2024-04-14 13:46:16.343837382 +0000 UTC m=+0.378109824 container init 25e416621d4036239dc419587544f348c677403acfb703c339ba78306ceb4e3e (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:46:16.372 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:16 smithi012 podman[60421]: 2024-04-14 13:46:16.348170802 +0000 UTC m=+0.382443246 container start 25e416621d4036239dc419587544f348c677403acfb703c339ba78306ceb4e3e (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-04-14T13:46:16.372 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:16 smithi012 ceph-osd[60440]: -- 172.21.15.12:0/3688728206 <== mon.1 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5592665cf680 con 0x559266599000 2024-04-14T13:46:16.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 270531 10.23 MB/sec execute 530 sec latency 34.979 ms 2024-04-14T13:46:16.800 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:16 smithi012 bash[60421]: 25e416621d4036239dc419587544f348c677403acfb703c339ba78306ceb4e3e 2024-04-14T13:46:16.801 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:16 smithi012 systemd[1]: Started Ceph osd.5 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:46:17.301 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:16 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:46:16.893+0000 7f6e2c8d47c0 -1 Falling back to public interface 2024-04-14T13:46:17.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 270801 10.22 MB/sec execute 531 sec latency 51.883 ms 2024-04-14T13:46:17.714 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:17 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:17.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:17 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:17.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:17 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:46:17.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:17 smithi008 ceph-mon[67070]: pgmap v231: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.8 MiB/s wr, 483 op/s; 177/1665 objects degraded (10.631%) 2024-04-14T13:46:17.729 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:17 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:17.729 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:17 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:17.729 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:17 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:46:17.729 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:17 smithi012 ceph-mon[53289]: pgmap v231: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.8 MiB/s wr, 483 op/s; 177/1665 objects degraded (10.631%) 2024-04-14T13:46:18.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 271136 10.21 MB/sec execute 532 sec latency 33.903 ms 2024-04-14T13:46:19.279 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:19 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:19.279 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:19 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:46:19.279 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:19 smithi008 ceph-mon[67070]: pgmap v232: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 333 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.9 MiB/s wr, 295 op/s; 176/1662 objects degraded (10.590%) 2024-04-14T13:46:19.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:19 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:19.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:19 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:46:19.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:19 smithi012 ceph-mon[53289]: pgmap v232: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 333 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.9 MiB/s wr, 295 op/s; 176/1662 objects degraded (10.590%) 2024-04-14T13:46:19.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 271520 10.21 MB/sec execute 533 sec latency 50.247 ms 2024-04-14T13:46:20.295 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:20 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 176/1662 objects degraded (10.590%), 22 pgs degraded (PG_DEGRADED) 2024-04-14T13:46:20.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 271941 10.21 MB/sec execute 534 sec latency 40.739 ms 2024-04-14T13:46:20.494 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:20 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 176/1662 objects degraded (10.590%), 22 pgs degraded (PG_DEGRADED) 2024-04-14T13:46:21.315 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:21 smithi008 ceph-mon[67070]: pgmap v233: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 346 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 5.3 MiB/s wr, 549 op/s; 190/1758 objects degraded (10.808%) 2024-04-14T13:46:21.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 272217 10.20 MB/sec execute 535 sec latency 61.667 ms 2024-04-14T13:46:21.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:21 smithi012 ceph-mon[53289]: pgmap v233: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 346 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 5.3 MiB/s wr, 549 op/s; 190/1758 objects degraded (10.808%) 2024-04-14T13:46:22.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:22 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:22.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:22 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:22.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 272553 10.20 MB/sec execute 536 sec latency 65.080 ms 2024-04-14T13:46:22.452 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:22 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:22.452 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:22 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:23.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 272800 10.19 MB/sec execute 537 sec latency 68.602 ms 2024-04-14T13:46:23.635 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:23 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:23.635 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:23 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:23.635 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:23 smithi008 ceph-mon[67070]: pgmap v234: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 345 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.7 MiB/s wr, 489 op/s; 190/1728 objects degraded (10.995%) 2024-04-14T13:46:23.734 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:23 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:23.735 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:23 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:23.735 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:23 smithi012 ceph-mon[53289]: pgmap v234: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 345 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.7 MiB/s wr, 489 op/s; 190/1728 objects degraded (10.995%) 2024-04-14T13:46:24.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 273188 10.19 MB/sec execute 538 sec latency 43.863 ms 2024-04-14T13:46:24.948 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:24 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 177/1611 objects degraded (10.987%), 22 pgs degraded (PG_DEGRADED) 2024-04-14T13:46:25.043 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:24 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 177/1611 objects degraded (10.987%), 22 pgs degraded (PG_DEGRADED) 2024-04-14T13:46:25.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 273524 10.20 MB/sec execute 539 sec latency 39.066 ms 2024-04-14T13:46:26.025 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:25 smithi008 ceph-mon[67070]: pgmap v235: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 344 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.4 MiB/s wr, 590 op/s; 177/1611 objects degraded (10.987%) 2024-04-14T13:46:26.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:25 smithi012 ceph-mon[53289]: pgmap v235: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 344 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.4 MiB/s wr, 590 op/s; 177/1611 objects degraded (10.987%) 2024-04-14T13:46:26.051 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:25 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:46:25.853+0000 7f6e2c8d47c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-04-14T13:46:26.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 273853 10.21 MB/sec execute 540 sec latency 35.318 ms 2024-04-14T13:46:27.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 274147 10.20 MB/sec execute 541 sec latency 53.516 ms 2024-04-14T13:46:27.514 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:27 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:46:27.207+0000 7f6e2c8d47c0 -1 osd.5 125 log_to_monitors true 2024-04-14T13:46:27.557 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:27 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:27.557 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:27 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:27.557 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:27 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:46:27.557 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:27 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:46:27.557 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:27 smithi008 ceph-mon[67070]: pgmap v236: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 453 op/s; 172/1608 objects degraded (10.697%) 2024-04-14T13:46:27.557 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:27 smithi008 ceph-mon[67070]: pgmap v237: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.9 MiB/s wr, 508 op/s; 172/1608 objects degraded (10.697%) 2024-04-14T13:46:27.557 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:27 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:27.557 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:27 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:27.558 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:27 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:46:27.558 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:27 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:46:27.558 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:27 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:46:27.558 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:27 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:46:27.558 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:27 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T13:46:27.558 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:27 smithi008 ceph-mon[67070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T13:46:27.558 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:27 smithi008 ceph-mon[67070]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-14T13:46:27.558 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:27 smithi008 ceph-mon[67070]: from='osd.5 [v2:172.21.15.12:6808/3354617740,v1:172.21.15.12:6809/3354617740]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T13:46:27.558 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:27 smithi008 ceph-mon[67070]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T13:46:27.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:27 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:27.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:27 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:27.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:27 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:46:27.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:27 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:46:27.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:27 smithi012 ceph-mon[53289]: pgmap v236: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 453 op/s; 172/1608 objects degraded (10.697%) 2024-04-14T13:46:27.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:27 smithi012 ceph-mon[53289]: pgmap v237: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.9 MiB/s wr, 508 op/s; 172/1608 objects degraded (10.697%) 2024-04-14T13:46:27.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:27 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:27.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:27 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:27.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:27 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:46:27.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:27 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:46:27.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:27 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:46:27.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:27 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:46:27.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:27 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T13:46:27.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:27 smithi012 ceph-mon[53289]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T13:46:27.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:27 smithi012 ceph-mon[53289]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-14T13:46:27.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:27 smithi012 ceph-mon[53289]: from='osd.5 [v2:172.21.15.12:6808/3354617740,v1:172.21.15.12:6809/3354617740]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T13:46:27.802 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:27 smithi012 ceph-mon[53289]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T13:46:28.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 274333 10.19 MB/sec execute 542 sec latency 124.998 ms 2024-04-14T13:46:28.550 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:46:28 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:46:28.060+0000 7f6e19d24700 -1 osd.5 125 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:46:28.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:28 smithi008 ceph-mon[67070]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-14T13:46:28.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:28 smithi008 ceph-mon[67070]: osdmap e128: 8 total, 7 up, 8 in 2024-04-14T13:46:28.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:28 smithi008 ceph-mon[67070]: from='osd.5 [v2:172.21.15.12:6808/3354617740,v1:172.21.15.12:6809/3354617740]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:46:28.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:28 smithi008 ceph-mon[67070]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:46:28.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:28 smithi008 ceph-mon[67070]: from='osd.5 ' entity='osd.5' 2024-04-14T13:46:28.626 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:46:29.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:28 smithi012 ceph-mon[53289]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-14T13:46:29.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:28 smithi012 ceph-mon[53289]: osdmap e128: 8 total, 7 up, 8 in 2024-04-14T13:46:29.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:28 smithi012 ceph-mon[53289]: from='osd.5 [v2:172.21.15.12:6808/3354617740,v1:172.21.15.12:6809/3354617740]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:46:29.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:28 smithi012 ceph-mon[53289]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:46:29.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:28 smithi012 ceph-mon[53289]: from='osd.5 ' entity='osd.5' 2024-04-14T13:46:29.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 274537 10.18 MB/sec execute 543 sec latency 296.555 ms 2024-04-14T13:46:29.602 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:46:29.602 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (8m) 86s ago 17m 15.3M - 0.23.0 ba2b418f427c cfddb9a7f858 2024-04-14T13:46:29.603 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (4m) 86s ago 17m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 1e69eabdcc36 2024-04-14T13:46:29.603 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (4m) 8s ago 16m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 60c7a2992eb0 2024-04-14T13:46:29.603 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 86s ago 17m - - 2024-04-14T13:46:29.603 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (11m) 86s ago 11m 39.0M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:46:29.603 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (11m) 8s ago 11m 15.1M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:46:29.603 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (7m) 86s ago 18m 508M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:46:29.603 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (6m) 8s ago 16m 453M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:46:29.603 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (5m) 86s ago 18m 49.3M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 673dd6f4c8d8 2024-04-14T13:46:29.603 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (5m) 8s ago 16m 43.5M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 d498dec29c4f 2024-04-14T13:46:29.603 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (11m) 86s ago 11m 284M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:46:29.603 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (11m) 8s ago 11m 91.0M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:46:29.603 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (7m) 86s ago 17m 11.4M - 1.3.1 1dbe0e931976 646dfca70be4 2024-04-14T13:46:29.603 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (7m) 8s ago 16m 19.0M - 1.3.1 1dbe0e931976 0a60dcd4044c 2024-04-14T13:46:29.603 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (3m) 86s ago 15m 326M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 29997ccc66ac 2024-04-14T13:46:29.603 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (2m) 86s ago 15m 272M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 97df88f10bd6 2024-04-14T13:46:29.604 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (2m) 86s ago 15m 205M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 9325b527c16c 2024-04-14T13:46:29.604 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (95s) 86s ago 14m 14.7M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 6868b5076d61 2024-04-14T13:46:29.604 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (51s) 8s ago 14m 274M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 178f9462f5ba 2024-04-14T13:46:29.604 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (13s) 8s ago 13m 14.8M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 25e416621d40 2024-04-14T13:46:29.604 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (13m) 8s ago 13m 579M 2705M 17.2.0 e1d6a67b021e 1db0e06c0707 2024-04-14T13:46:29.604 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (12m) 8s ago 12m 612M 2705M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:46:29.604 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (6m) 86s ago 16m 69.9M - 2.33.4 514e6a882f6e 12f97f75d620 2024-04-14T13:46:29.624 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:29 smithi008 ceph-mon[67070]: pgmap v239: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 338 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.9 MiB/s wr, 270 op/s; 175/1632 objects degraded (10.723%) 2024-04-14T13:46:29.625 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:29 smithi008 ceph-mon[67070]: from='client.44305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:46:29.625 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:29 smithi008 ceph-mon[67070]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T13:46:29.625 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:29 smithi008 ceph-mon[67070]: osd.5 [v2:172.21.15.12:6808/3354617740,v1:172.21.15.12:6809/3354617740] boot 2024-04-14T13:46:29.625 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:29 smithi008 ceph-mon[67070]: osdmap e129: 8 total, 8 up, 8 in 2024-04-14T13:46:29.625 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:46:30.047 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:46:30.047 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:46:30.048 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:46:30.048 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:46:30.048 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:46:30.048 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:46:30.048 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:46:30.048 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:46:30.048 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-14T13:46:30.048 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 6 2024-04-14T13:46:30.048 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:46:30.048 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:46:30.048 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:46:30.048 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:46:30.048 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:46:30.048 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-14T13:46:30.049 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 10 2024-04-14T13:46:30.049 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:46:30.049 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:46:30.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:29 smithi012 ceph-mon[53289]: pgmap v239: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 338 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.9 MiB/s wr, 270 op/s; 175/1632 objects degraded (10.723%) 2024-04-14T13:46:30.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:29 smithi012 ceph-mon[53289]: from='client.44305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:46:30.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:29 smithi012 ceph-mon[53289]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T13:46:30.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:29 smithi012 ceph-mon[53289]: osd.5 [v2:172.21.15.12:6808/3354617740,v1:172.21.15.12:6809/3354617740] boot 2024-04-14T13:46:30.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:29 smithi012 ceph-mon[53289]: osdmap e129: 8 total, 8 up, 8 in 2024-04-14T13:46:30.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:46:30.377 INFO:teuthology.orchestra.run.smithi008.stdout: 5 274626 10.16 MB/sec execute 544 sec latency 1235.922 ms 2024-04-14T13:46:30.506 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:46:30.506 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:46:30.506 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:46:30.506 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:46:30.506 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-14T13:46:30.506 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-04-14T13:46:30.507 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-14T13:46:30.507 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-04-14T13:46:30.507 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:46:30.507 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "12/23 daemons upgraded", 2024-04-14T13:46:30.507 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T13:46:30.507 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:46:30.507 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:46:30.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:30 smithi008 ceph-mon[67070]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:46:30.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:30 smithi008 ceph-mon[67070]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:46:30.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:30 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/2521930115' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:46:30.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:30 smithi008 ceph-mon[67070]: osdmap e130: 8 total, 8 up, 8 in 2024-04-14T13:46:30.924 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 175/1632 objects degraded (10.723%), 22 pgs degraded 2024-04-14T13:46:30.924 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:46:30.924 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-14T13:46:30.924 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 175/1632 objects degraded (10.723%), 22 pgs degraded 2024-04-14T13:46:30.924 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.2 is active+undersized+degraded, acting [1,6] 2024-04-14T13:46:30.924 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.3 is active+undersized+degraded, acting [2,7] 2024-04-14T13:46:30.924 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.b is active+undersized+degraded, acting [7,4] 2024-04-14T13:46:30.924 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.c is active+undersized+degraded, acting [2,0] 2024-04-14T13:46:30.924 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.12 is active+undersized+degraded, acting [3,7] 2024-04-14T13:46:30.924 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.14 is active+undersized+degraded, acting [6,3] 2024-04-14T13:46:30.924 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.15 is active+undersized+degraded, acting [1,0] 2024-04-14T13:46:30.924 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.16 is active+undersized+degraded, acting [6,2] 2024-04-14T13:46:30.925 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.17 is active+undersized+degraded, acting [6,2] 2024-04-14T13:46:30.925 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.18 is active+undersized+degraded, acting [4,7] 2024-04-14T13:46:30.925 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1c is active+undersized+degraded, acting [7,2] 2024-04-14T13:46:30.925 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-04-14T13:46:30.925 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2024-04-14T13:46:30.925 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-04-14T13:46:30.925 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2024-04-14T13:46:30.925 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.d is active+undersized+degraded, acting [7,6] 2024-04-14T13:46:30.925 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.10 is active+undersized+degraded, acting [6,0] 2024-04-14T13:46:30.925 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-04-14T13:46:30.925 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-04-14T13:46:30.925 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2024-04-14T13:46:30.925 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2024-04-14T13:46:30.925 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-04-14T13:46:31.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:30 smithi012 ceph-mon[53289]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:46:31.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:30 smithi012 ceph-mon[53289]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:46:31.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:30 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/2521930115' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:46:31.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:30 smithi012 ceph-mon[53289]: osdmap e130: 8 total, 8 up, 8 in 2024-04-14T13:46:31.383 INFO:teuthology.orchestra.run.smithi008.stdout: 5 274649 10.14 MB/sec execute 545 sec latency 2126.406 ms 2024-04-14T13:46:31.752 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:31 smithi008 ceph-mon[67070]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:46:31.752 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:31 smithi008 ceph-mon[67070]: pgmap v242: 97 pgs: 1 remapped+peering, 10 peering, 5 active+undersized, 13 active+undersized+degraded, 68 active+clean; 345 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 6.1 MiB/s wr, 517 op/s; 96/1746 objects degraded (5.498%) 2024-04-14T13:46:31.752 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:31 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/2234991814' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:46:31.753 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:31 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 96/1746 objects degraded (5.498%), 13 pgs degraded (PG_DEGRADED) 2024-04-14T13:46:31.753 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:31 smithi008 ceph-mon[67070]: osdmap e131: 8 total, 8 up, 8 in 2024-04-14T13:46:32.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:31 smithi012 ceph-mon[53289]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:46:32.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:31 smithi012 ceph-mon[53289]: pgmap v242: 97 pgs: 1 remapped+peering, 10 peering, 5 active+undersized, 13 active+undersized+degraded, 68 active+clean; 345 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 6.1 MiB/s wr, 517 op/s; 96/1746 objects degraded (5.498%) 2024-04-14T13:46:32.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:31 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/2234991814' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:46:32.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:31 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 96/1746 objects degraded (5.498%), 13 pgs degraded (PG_DEGRADED) 2024-04-14T13:46:32.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:31 smithi012 ceph-mon[53289]: osdmap e131: 8 total, 8 up, 8 in 2024-04-14T13:46:32.383 INFO:teuthology.orchestra.run.smithi008.stdout: 5 274908 10.13 MB/sec execute 546 sec latency 270.110 ms 2024-04-14T13:46:33.349 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:33 smithi008 ceph-mon[67070]: osdmap e132: 8 total, 8 up, 8 in 2024-04-14T13:46:33.349 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:33 smithi008 ceph-mon[67070]: pgmap v245: 97 pgs: 1 remapped+peering, 10 peering, 3 active+undersized, 1 active+undersized+remapped, 10 active+undersized+degraded, 72 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 5.5 MiB/s wr, 502 op/s; 66/1713 objects degraded (3.853%); 212 KiB/s, 13 objects/s recovering 2024-04-14T13:46:33.383 INFO:teuthology.orchestra.run.smithi008.stdout: 5 275115 10.13 MB/sec execute 547 sec latency 1030.534 ms 2024-04-14T13:46:33.451 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:33 smithi012 ceph-mon[53289]: osdmap e132: 8 total, 8 up, 8 in 2024-04-14T13:46:33.452 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:33 smithi012 ceph-mon[53289]: pgmap v245: 97 pgs: 1 remapped+peering, 10 peering, 3 active+undersized, 1 active+undersized+remapped, 10 active+undersized+degraded, 72 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 5.5 MiB/s wr, 502 op/s; 66/1713 objects degraded (3.853%); 212 KiB/s, 13 objects/s recovering 2024-04-14T13:46:34.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:34 smithi008 ceph-mon[67070]: osdmap e133: 8 total, 8 up, 8 in 2024-04-14T13:46:34.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:34 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:34.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:34 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:46:34.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 275482 10.12 MB/sec execute 548 sec latency 59.470 ms 2024-04-14T13:46:34.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:34 smithi012 ceph-mon[53289]: osdmap e133: 8 total, 8 up, 8 in 2024-04-14T13:46:34.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:34 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:34.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:34 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:46:35.109 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:35 smithi008 ceph-mon[67070]: pgmap v247: 97 pgs: 5 peering, 92 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 2.4 MiB/s wr, 302 op/s; 1.5 MiB/s, 45 objects/s recovering 2024-04-14T13:46:35.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 275970 10.12 MB/sec execute 549 sec latency 43.928 ms 2024-04-14T13:46:35.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:35 smithi012 ceph-mon[53289]: pgmap v247: 97 pgs: 5 peering, 92 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 2.4 MiB/s wr, 302 op/s; 1.5 MiB/s, 45 objects/s recovering 2024-04-14T13:46:36.126 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:36 smithi008 ceph-mon[67070]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 66/1713 objects degraded (3.853%), 10 pgs degraded) 2024-04-14T13:46:36.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 276241 10.11 MB/sec execute 550 sec latency 44.382 ms 2024-04-14T13:46:36.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:36 smithi012 ceph-mon[53289]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 66/1713 objects degraded (3.853%), 10 pgs degraded) 2024-04-14T13:46:37.179 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:37 smithi008 ceph-mon[67070]: pgmap v248: 97 pgs: 2 peering, 95 active+clean; 333 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 1.9 MiB/s wr, 250 op/s; 1.3 MiB/s, 40 objects/s recovering 2024-04-14T13:46:37.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 276625 10.12 MB/sec execute 551 sec latency 57.933 ms 2024-04-14T13:46:37.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:37 smithi012 ceph-mon[53289]: pgmap v248: 97 pgs: 2 peering, 95 active+clean; 333 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 1.9 MiB/s wr, 250 op/s; 1.3 MiB/s, 40 objects/s recovering 2024-04-14T13:46:38.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 276934 10.12 MB/sec execute 552 sec latency 54.654 ms 2024-04-14T13:46:39.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 277324 10.14 MB/sec execute 553 sec latency 65.152 ms 2024-04-14T13:46:39.755 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:39 smithi008 ceph-mon[67070]: pgmap v249: 97 pgs: 97 active+clean; 329 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.8 MiB/s wr, 233 op/s; 1.1 MiB/s, 32 objects/s recovering 2024-04-14T13:46:40.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:39 smithi012 ceph-mon[53289]: pgmap v249: 97 pgs: 97 active+clean; 329 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.8 MiB/s wr, 233 op/s; 1.1 MiB/s, 32 objects/s recovering 2024-04-14T13:46:40.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 277557 10.13 MB/sec execute 554 sec latency 48.545 ms 2024-04-14T13:46:41.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 277822 10.13 MB/sec execute 555 sec latency 43.246 ms 2024-04-14T13:46:41.795 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:41 smithi008 ceph-mon[67070]: pgmap v250: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.6 MiB/s wr, 476 op/s; 881 KiB/s, 22 objects/s recovering 2024-04-14T13:46:42.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:41 smithi012 ceph-mon[53289]: pgmap v250: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.6 MiB/s wr, 476 op/s; 881 KiB/s, 22 objects/s recovering 2024-04-14T13:46:42.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 278150 10.11 MB/sec execute 556 sec latency 50.591 ms 2024-04-14T13:46:42.761 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:42 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T13:46:42.761 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:42 smithi012 ceph-mon[53289]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T13:46:42.761 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:42 smithi012 ceph-mon[53289]: Upgrade: osd.6 is safe to restart 2024-04-14T13:46:42.818 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:42 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T13:46:42.818 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:42 smithi008 ceph-mon[67070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T13:46:42.818 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:42 smithi008 ceph-mon[67070]: Upgrade: osd.6 is safe to restart 2024-04-14T13:46:43.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 278410 10.10 MB/sec execute 557 sec latency 87.464 ms 2024-04-14T13:46:43.838 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:43 smithi008 ceph-mon[67070]: pgmap v251: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.0 MiB/s wr, 408 op/s; 743 KiB/s, 18 objects/s recovering 2024-04-14T13:46:44.040 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:43 smithi012 ceph-mon[53289]: pgmap v251: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.0 MiB/s wr, 408 op/s; 743 KiB/s, 18 objects/s recovering 2024-04-14T13:46:44.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 278738 10.11 MB/sec execute 558 sec latency 44.817 ms 2024-04-14T13:46:45.120 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:45 smithi008 ceph-mon[67070]: Upgrade: Updating osd.6 2024-04-14T13:46:45.120 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:45 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:45.120 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:45 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T13:46:45.120 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:45 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:46:45.120 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:45 smithi008 ceph-mon[67070]: Deploying daemon osd.6 on smithi012 2024-04-14T13:46:45.120 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:45 smithi008 ceph-mon[67070]: pgmap v252: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.4 MiB/s wr, 422 op/s; 569 KiB/s, 8 objects/s recovering 2024-04-14T13:46:45.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 279154 10.10 MB/sec execute 559 sec latency 45.289 ms 2024-04-14T13:46:45.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:45 smithi012 ceph-mon[53289]: Upgrade: Updating osd.6 2024-04-14T13:46:45.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:45 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:45.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:45 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T13:46:45.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:45 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:46:45.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:45 smithi012 ceph-mon[53289]: Deploying daemon osd.6 on smithi012 2024-04-14T13:46:45.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:45 smithi012 ceph-mon[53289]: pgmap v252: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.4 MiB/s wr, 422 op/s; 569 KiB/s, 8 objects/s recovering 2024-04-14T13:46:46.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 279614 10.09 MB/sec execute 560 sec latency 31.020 ms 2024-04-14T13:46:47.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 279878 10.08 MB/sec execute 561 sec latency 67.412 ms 2024-04-14T13:46:47.674 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:47 smithi008 ceph-mon[67070]: pgmap v253: 97 pgs: 97 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.5 MiB/s wr, 449 op/s; 120 KiB/s, 3 objects/s recovering 2024-04-14T13:46:47.967 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:47 smithi012 ceph-mon[53289]: pgmap v253: 97 pgs: 97 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.5 MiB/s wr, 449 op/s; 120 KiB/s, 3 objects/s recovering 2024-04-14T13:46:48.275 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:47 smithi012 systemd[1]: Stopping Ceph osd.6 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:46:48.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 280258 10.10 MB/sec execute 562 sec latency 49.370 ms 2024-04-14T13:46:48.551 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:48 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[37415]: 2024-04-14T13:46:48.282+0000 7f3313026700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:46:48.551 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:48 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[37415]: 2024-04-14T13:46:48.282+0000 7f3313026700 -1 osd.6 133 *** Got signal Terminated *** 2024-04-14T13:46:48.551 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:48 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[37415]: 2024-04-14T13:46:48.282+0000 7f3313026700 -1 osd.6 133 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T13:46:49.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:49.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:46:49.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:49.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:49 smithi008 ceph-mon[67070]: osd.6 marked itself down and dead 2024-04-14T13:46:49.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:49 smithi008 ceph-mon[67070]: pgmap v254: 97 pgs: 97 active+clean; 341 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.4 MiB/s wr, 453 op/s 2024-04-14T13:46:49.361 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:49.362 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:46:49.362 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:49.362 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:49 smithi012 ceph-mon[53289]: osd.6 marked itself down and dead 2024-04-14T13:46:49.362 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:49 smithi012 ceph-mon[53289]: pgmap v254: 97 pgs: 97 active+clean; 341 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.4 MiB/s wr, 453 op/s 2024-04-14T13:46:49.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 280646 10.11 MB/sec execute 563 sec latency 78.902 ms 2024-04-14T13:46:50.177 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:50 smithi012 ceph-mon[53289]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T13:46:50.177 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:50 smithi012 ceph-mon[53289]: osdmap e134: 8 total, 7 up, 8 in 2024-04-14T13:46:50.178 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:49 smithi012 podman[62229]: 2024-04-14 13:46:49.899738214 +0000 UTC m=+1.740977001 container died 1db0e06c0707b21fff762aa510983549429cf2dcb6071f100f5bf35090520084 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, vcs-type=git, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, ceph=True, io.openshift.expose-services=, GIT_CLEAN=True, name=centos-stream, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc.) 2024-04-14T13:46:50.279 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:50 smithi008 ceph-mon[67070]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T13:46:50.279 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:50 smithi008 ceph-mon[67070]: osdmap e134: 8 total, 7 up, 8 in 2024-04-14T13:46:50.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 280809 10.10 MB/sec execute 564 sec latency 944.234 ms 2024-04-14T13:46:50.550 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:50 smithi012 podman[62229]: 2024-04-14 13:46:50.373252347 +0000 UTC m=+2.214491144 container cleanup 1db0e06c0707b21fff762aa510983549429cf2dcb6071f100f5bf35090520084 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, name=centos-stream, distribution-scope=public, GIT_BRANCH=HEAD, vcs-type=git, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, CEPH_POINT_RELEASE=-17.2.0) 2024-04-14T13:46:50.551 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:50 smithi012 bash[62229]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6 2024-04-14T13:46:51.051 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:50 smithi012 podman[62256]: 2024-04-14 13:46:50.708805629 +0000 UTC m=+0.806316720 container remove 1db0e06c0707b21fff762aa510983549429cf2dcb6071f100f5bf35090520084 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, version=8, io.openshift.tags=base centos centos-stream, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, ceph=True, build-date=2022-05-03T08:36:31.336870, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , distribution-scope=public, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream) 2024-04-14T13:46:51.254 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:51 smithi008 ceph-mon[67070]: osdmap e135: 8 total, 7 up, 8 in 2024-04-14T13:46:51.254 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:51 smithi008 ceph-mon[67070]: pgmap v257: 97 pgs: 17 peering, 4 stale+active+clean, 76 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.5 MiB/s wr, 533 op/s 2024-04-14T13:46:51.365 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:51 smithi012 ceph-mon[53289]: osdmap e135: 8 total, 7 up, 8 in 2024-04-14T13:46:51.366 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:51 smithi012 ceph-mon[53289]: pgmap v257: 97 pgs: 17 peering, 4 stale+active+clean, 76 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.5 MiB/s wr, 533 op/s 2024-04-14T13:46:51.366 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:51 smithi012 podman[62348]: 2024-04-14 13:46:50.981437103 +0000 UTC m=+0.020466190 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:46:51.366 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:51 smithi012 podman[62348]: 2024-04-14 13:46:51.101508602 +0000 UTC m=+0.140537683 container create ce2e9597be5fcd7ce5eb90d4ce3d8529e39b67ef35126407b54c12c999f96aee (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=main-a971fd5, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T13:46:51.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 281130 10.11 MB/sec execute 565 sec latency 45.811 ms 2024-04-14T13:46:51.718 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:51 smithi012 podman[62348]: 2024-04-14 13:46:51.363110009 +0000 UTC m=+0.402139088 container init ce2e9597be5fcd7ce5eb90d4ce3d8529e39b67ef35126407b54c12c999f96aee (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T13:46:51.718 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:51 smithi012 podman[62348]: 2024-04-14 13:46:51.367912301 +0000 UTC m=+0.406941389 container start ce2e9597be5fcd7ce5eb90d4ce3d8529e39b67ef35126407b54c12c999f96aee (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-14T13:46:51.718 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:51 smithi012 podman[62348]: 2024-04-14 13:46:51.471371307 +0000 UTC m=+0.510400403 container attach ce2e9597be5fcd7ce5eb90d4ce3d8529e39b67ef35126407b54c12c999f96aee (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-a971fd5) 2024-04-14T13:46:51.718 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:51 smithi012 podman[62348]: 2024-04-14 13:46:51.715244023 +0000 UTC m=+0.754273106 container died ce2e9597be5fcd7ce5eb90d4ce3d8529e39b67ef35126407b54c12c999f96aee (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-14T13:46:52.277 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:52 smithi008 ceph-mon[67070]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-14T13:46:52.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 281324 10.10 MB/sec execute 566 sec latency 41.197 ms 2024-04-14T13:46:52.479 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:52 smithi012 ceph-mon[53289]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-14T13:46:52.801 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:52 smithi012 podman[62374]: 2024-04-14 13:46:52.477010443 +0000 UTC m=+0.755145852 container remove ce2e9597be5fcd7ce5eb90d4ce3d8529e39b67ef35126407b54c12c999f96aee (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True) 2024-04-14T13:46:52.802 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:52 smithi012 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.6.service: Deactivated successfully. 2024-04-14T13:46:52.802 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:52 smithi012 systemd[1]: Stopped Ceph osd.6 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:46:52.802 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:52 smithi012 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.6.service: Consumed 5min 19.283s CPU time. 2024-04-14T13:46:52.802 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:52 smithi012 systemd[1]: Starting Ceph osd.6 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:46:53.247 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:52 smithi012 podman[62476]: 2024-04-14 13:46:52.833990769 +0000 UTC m=+0.021026177 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:46:53.247 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:52 smithi012 podman[62476]: 2024-04-14 13:46:52.953257635 +0000 UTC m=+0.140293044 container create 693c009202d8d1d99986d018d8eaebbc6ed59293585495c686d63a2137ba9f0a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-activate, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T13:46:53.289 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:53 smithi008 ceph-mon[67070]: pgmap v258: 97 pgs: 1 active+undersized, 2 active+undersized+degraded, 17 peering, 4 stale+active+clean, 73 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 5.0 MiB/s wr, 582 op/s; 25/1683 objects degraded (1.485%) 2024-04-14T13:46:53.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 281605 10.08 MB/sec execute 567 sec latency 49.705 ms 2024-04-14T13:46:53.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:53 smithi012 ceph-mon[53289]: pgmap v258: 97 pgs: 1 active+undersized, 2 active+undersized+degraded, 17 peering, 4 stale+active+clean, 73 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 5.0 MiB/s wr, 582 op/s; 25/1683 objects degraded (1.485%) 2024-04-14T13:46:54.051 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:53 smithi012 podman[62476]: 2024-04-14 13:46:53.723192024 +0000 UTC m=+0.910227432 container init 693c009202d8d1d99986d018d8eaebbc6ed59293585495c686d63a2137ba9f0a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-activate, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS) 2024-04-14T13:46:54.051 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:53 smithi012 podman[62476]: 2024-04-14 13:46:53.726783132 +0000 UTC m=+0.913818544 container start 693c009202d8d1d99986d018d8eaebbc6ed59293585495c686d63a2137ba9f0a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-activate, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-14T13:46:54.051 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:53 smithi012 podman[62476]: 2024-04-14 13:46:53.815090933 +0000 UTC m=+1.002126341 container attach 693c009202d8d1d99986d018d8eaebbc6ed59293585495c686d63a2137ba9f0a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-activate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-14T13:46:54.320 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:54 smithi008 ceph-mon[67070]: Health check failed: Degraded data redundancy: 25/1683 objects degraded (1.485%), 2 pgs degraded (PG_DEGRADED) 2024-04-14T13:46:54.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 281841 10.07 MB/sec execute 568 sec latency 54.838 ms 2024-04-14T13:46:54.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:54 smithi012 ceph-mon[53289]: Health check failed: Degraded data redundancy: 25/1683 objects degraded (1.485%), 2 pgs degraded (PG_DEGRADED) 2024-04-14T13:46:54.551 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:54 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-activate[62491]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-14T13:46:54.551 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:54 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-activate[62491]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-14T13:46:54.551 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:54 smithi012 bash[62476]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-14T13:46:54.551 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:54 smithi012 bash[62476]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-14T13:46:54.551 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:54 smithi012 bash[62476]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-14T13:46:54.551 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:54 smithi012 bash[62476]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-14T13:46:54.551 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:54 smithi012 bash[62476]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-14T13:46:54.551 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:54 smithi012 bash[62476]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-14T13:46:54.551 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:54 smithi012 bash[62476]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-14T13:46:54.552 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:54 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-activate[62491]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-14T13:46:54.552 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:54 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-activate[62491]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-14T13:46:54.552 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:54 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-activate[62491]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-14T13:46:54.552 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:54 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-activate[62491]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-14T13:46:54.552 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:54 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-activate[62491]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-14T13:46:54.552 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:54 smithi012 podman[62476]: 2024-04-14 13:46:54.297892028 +0000 UTC m=+1.484927436 container died 693c009202d8d1d99986d018d8eaebbc6ed59293585495c686d63a2137ba9f0a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-14T13:46:55.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:55 smithi012 ceph-mon[53289]: pgmap v259: 97 pgs: 5 active+undersized, 18 active+undersized+degraded, 12 peering, 62 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.9 MiB/s wr, 457 op/s; 136/1707 objects degraded (7.967%) 2024-04-14T13:46:55.301 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:55 smithi012 podman[62623]: 2024-04-14 13:46:55.029472475 +0000 UTC m=+0.725730998 container remove 693c009202d8d1d99986d018d8eaebbc6ed59293585495c686d63a2137ba9f0a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-a971fd5) 2024-04-14T13:46:55.301 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:55 smithi012 podman[62684]: 2024-04-14 13:46:55.164295166 +0000 UTC m=+0.022152775 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:46:55.301 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:55 smithi012 podman[62684]: 2024-04-14 13:46:55.272338538 +0000 UTC m=+0.130196150 container create 97f0b691cf873bce2a3869deabb10117219eb6b300a1d0bc999b4b8a128ea11a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T13:46:55.359 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:55 smithi008 ceph-mon[67070]: pgmap v259: 97 pgs: 5 active+undersized, 18 active+undersized+degraded, 12 peering, 62 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.9 MiB/s wr, 457 op/s; 136/1707 objects degraded (7.967%) 2024-04-14T13:46:55.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 282147 10.07 MB/sec execute 569 sec latency 38.548 ms 2024-04-14T13:46:55.801 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:55 smithi012 podman[62684]: 2024-04-14 13:46:55.542089141 +0000 UTC m=+0.399946754 container init 97f0b691cf873bce2a3869deabb10117219eb6b300a1d0bc999b4b8a128ea11a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-04-14T13:46:55.801 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:55 smithi012 podman[62684]: 2024-04-14 13:46:55.546824597 +0000 UTC m=+0.404682209 container start 97f0b691cf873bce2a3869deabb10117219eb6b300a1d0bc999b4b8a128ea11a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240403, io.buildah.version=1.33.5) 2024-04-14T13:46:55.801 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:55 smithi012 ceph-osd[62701]: -- 172.21.15.12:0/107430463 <== mon.1 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562f74f37680 con 0x562f74f00c00 2024-04-14T13:46:55.801 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:55 smithi012 bash[62684]: 97f0b691cf873bce2a3869deabb10117219eb6b300a1d0bc999b4b8a128ea11a 2024-04-14T13:46:55.801 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:55 smithi012 systemd[1]: Started Ceph osd.6 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:46:56.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 282539 10.07 MB/sec execute 570 sec latency 67.900 ms 2024-04-14T13:46:56.551 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:46:56 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:46:56.085+0000 7f3b1d9637c0 -1 Falling back to public interface 2024-04-14T13:46:56.878 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:56 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:56.878 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:56 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:56.878 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:56 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:46:57.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:56 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:56 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:46:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:56 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:46:57.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 282964 10.06 MB/sec execute 571 sec latency 81.109 ms 2024-04-14T13:46:57.920 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:57 smithi008 ceph-mon[67070]: pgmap v260: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.5 MiB/s wr, 497 op/s; 193/1761 objects degraded (10.960%) 2024-04-14T13:46:57.920 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:57 smithi008 ceph-mon[67070]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-14T13:46:58.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:57 smithi012 ceph-mon[53289]: pgmap v260: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.5 MiB/s wr, 497 op/s; 193/1761 objects degraded (10.960%) 2024-04-14T13:46:58.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:57 smithi012 ceph-mon[53289]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-14T13:46:58.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 283333 10.05 MB/sec execute 572 sec latency 52.285 ms 2024-04-14T13:46:58.945 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:58 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 190/1737 objects degraded (10.938%), 25 pgs degraded (PG_DEGRADED) 2024-04-14T13:46:59.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:58 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 190/1737 objects degraded (10.938%), 25 pgs degraded (PG_DEGRADED) 2024-04-14T13:46:59.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 283590 10.05 MB/sec execute 573 sec latency 34.652 ms 2024-04-14T13:46:59.963 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:46:59 smithi008 ceph-mon[67070]: pgmap v261: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.4 MiB/s wr, 458 op/s; 190/1737 objects degraded (10.938%) 2024-04-14T13:47:00.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:46:59 smithi012 ceph-mon[53289]: pgmap v261: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.4 MiB/s wr, 458 op/s; 190/1737 objects degraded (10.938%) 2024-04-14T13:47:00.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 284122 10.07 MB/sec execute 574 sec latency 38.619 ms 2024-04-14T13:47:01.253 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:01 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:01.253 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:01 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:01.253 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:01 smithi008 ceph-mon[67070]: pgmap v262: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 538 op/s; 176/1650 objects degraded (10.667%) 2024-04-14T13:47:01.374 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:47:01.384 INFO:teuthology.orchestra.run.smithi008.stdout: 5 284498 10.09 MB/sec execute 575 sec latency 48.772 ms 2024-04-14T13:47:01.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:01 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:01.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:01 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:01.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:01 smithi012 ceph-mon[53289]: pgmap v262: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 538 op/s; 176/1650 objects degraded (10.667%) 2024-04-14T13:47:02.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 284744 10.09 MB/sec execute 576 sec latency 34.605 ms 2024-04-14T13:47:02.427 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:47:02.427 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (8m) 119s ago 18m 15.3M - 0.23.0 ba2b418f427c cfddb9a7f858 2024-04-14T13:47:02.427 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (4m) 119s ago 17m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 1e69eabdcc36 2024-04-14T13:47:02.428 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (4m) 2s ago 17m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 60c7a2992eb0 2024-04-14T13:47:02.428 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 119s ago 17m - - 2024-04-14T13:47:02.428 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (12m) 119s ago 12m 39.0M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:47:02.428 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (12m) 2s ago 12m 15.2M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:47:02.428 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (7m) 119s ago 19m 508M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:47:02.428 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (6m) 2s ago 17m 453M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:47:02.428 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (6m) 119s ago 19m 49.3M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 673dd6f4c8d8 2024-04-14T13:47:02.428 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (5m) 2s ago 17m 44.4M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 d498dec29c4f 2024-04-14T13:47:02.428 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (11m) 119s ago 11m 284M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:47:02.428 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (11m) 2s ago 11m 91.0M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:47:02.428 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (8m) 119s ago 17m 11.4M - 1.3.1 1dbe0e931976 646dfca70be4 2024-04-14T13:47:02.428 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (7m) 2s ago 16m 19.3M - 1.3.1 1dbe0e931976 0a60dcd4044c 2024-04-14T13:47:02.428 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (4m) 119s ago 16m 326M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 29997ccc66ac 2024-04-14T13:47:02.428 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (3m) 119s ago 15m 272M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 97df88f10bd6 2024-04-14T13:47:02.428 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (2m) 119s ago 15m 205M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 9325b527c16c 2024-04-14T13:47:02.429 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (2m) 119s ago 15m 14.7M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 6868b5076d61 2024-04-14T13:47:02.429 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (84s) 2s ago 14m 319M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 178f9462f5ba 2024-04-14T13:47:02.429 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (46s) 2s ago 14m 262M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 25e416621d40 2024-04-14T13:47:02.429 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (7s) 2s ago 13m 11.5M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 97f0b691cf87 2024-04-14T13:47:02.429 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (13m) 2s ago 13m 666M 2705M 17.2.0 e1d6a67b021e 99bab39bb434 2024-04-14T13:47:02.429 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (6m) 119s ago 16m 69.9M - 2.33.4 514e6a882f6e 12f97f75d620 2024-04-14T13:47:02.771 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:02 smithi008 ceph-mon[67070]: from='client.34414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:47:02.771 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:02 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:02.771 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:02 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:02.771 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:02 smithi008 ceph-mon[67070]: from='client.34418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:47:02.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:02 smithi012 ceph-mon[53289]: from='client.34414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:47:02.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:02 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:02.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:02 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:02.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:02 smithi012 ceph-mon[53289]: from='client.34418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:47:03.018 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:47:03.019 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:47:03.019 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:47:03.019 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:47:03.019 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:47:03.019 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:47:03.019 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:47:03.019 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:47:03.019 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-14T13:47:03.019 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 6 2024-04-14T13:47:03.019 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:47:03.019 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:47:03.019 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:47:03.019 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:47:03.020 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:47:03.020 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-14T13:47:03.020 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 10 2024-04-14T13:47:03.020 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:47:03.020 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:47:03.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 284984 10.07 MB/sec execute 577 sec latency 32.529 ms 2024-04-14T13:47:03.517 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:47:03.518 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:47:03.518 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:47:03.518 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:47:03.518 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-14T13:47:03.518 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-04-14T13:47:03.518 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-14T13:47:03.518 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-04-14T13:47:03.518 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:47:03.518 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "13/23 daemons upgraded", 2024-04-14T13:47:03.518 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T13:47:03.518 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:47:03.518 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:47:03.765 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:03 smithi012 ceph-mon[53289]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:47:03.766 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:03 smithi012 ceph-mon[53289]: pgmap v263: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 340 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.2 MiB/s wr, 470 op/s; 177/1659 objects degraded (10.669%) 2024-04-14T13:47:03.766 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:03 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/595850402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:47:03.766 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:03.766 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:47:03.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:03 smithi008 ceph-mon[67070]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:47:03.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:03 smithi008 ceph-mon[67070]: pgmap v263: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 340 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.2 MiB/s wr, 470 op/s; 177/1659 objects degraded (10.669%) 2024-04-14T13:47:03.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:03 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/595850402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:47:03.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:03.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:47:04.126 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 177/1659 objects degraded (10.669%), 25 pgs degraded 2024-04-14T13:47:04.126 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:47:04.126 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-14T13:47:04.126 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-14T13:47:04.126 INFO:teuthology.orchestra.run.smithi008.stdout: osd.6 (root=default,host=smithi012) is down 2024-04-14T13:47:04.126 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 177/1659 objects degraded (10.669%), 25 pgs degraded 2024-04-14T13:47:04.127 INFO:teuthology.orchestra.run.smithi008.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-04-14T13:47:04.127 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-04-14T13:47:04.127 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.6 is active+undersized+degraded, acting [1,4] 2024-04-14T13:47:04.127 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.7 is active+undersized+degraded, acting [7,2] 2024-04-14T13:47:04.127 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-04-14T13:47:04.127 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.14 is active+undersized+degraded, acting [3,5] 2024-04-14T13:47:04.127 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.16 is active+undersized+degraded, acting [5,2] 2024-04-14T13:47:04.127 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-04-14T13:47:04.127 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.19 is active+undersized+degraded, acting [3,0] 2024-04-14T13:47:04.127 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1a is active+undersized+degraded, acting [4,7] 2024-04-14T13:47:04.127 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1b is active+undersized+degraded, acting [3,7] 2024-04-14T13:47:04.127 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1d is active+undersized+degraded, acting [7,0] 2024-04-14T13:47:04.127 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2024-04-14T13:47:04.128 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-04-14T13:47:04.128 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-04-14T13:47:04.128 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-04-14T13:47:04.128 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-04-14T13:47:04.128 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-04-14T13:47:04.128 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-04-14T13:47:04.128 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2024-04-14T13:47:04.128 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-04-14T13:47:04.128 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-04-14T13:47:04.128 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-04-14T13:47:04.128 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.5 is active+undersized+degraded, acting [4,7] 2024-04-14T13:47:04.128 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.a is active+undersized+degraded, acting [2,7] 2024-04-14T13:47:04.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 285292 10.06 MB/sec execute 578 sec latency 48.972 ms 2024-04-14T13:47:04.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:04 smithi008 ceph-mon[67070]: from='client.44331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:47:04.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:04 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/1699935272' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:47:04.813 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:04 smithi012 ceph-mon[53289]: from='client.44331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:47:04.813 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:04 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/1699935272' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:47:05.261 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:47:05 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:47:05.045+0000 7f3b1d9637c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-04-14T13:47:05.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 285581 10.05 MB/sec execute 579 sec latency 45.697 ms 2024-04-14T13:47:05.571 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:05 smithi008 ceph-mon[67070]: pgmap v264: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.0 MiB/s wr, 471 op/s; 182/1695 objects degraded (10.737%) 2024-04-14T13:47:05.571 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:05 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 182/1695 objects degraded (10.737%), 25 pgs degraded (PG_DEGRADED) 2024-04-14T13:47:05.571 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:05 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:05.571 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:05 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:05.571 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:05 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:47:05.571 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:05 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:47:05.571 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:05 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:05.571 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:05 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:05.571 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:05 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:47:05.572 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:05 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:47:05.572 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:05 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:47:05.572 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:05 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:47:05.572 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:05 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T13:47:06.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:05 smithi012 ceph-mon[53289]: pgmap v264: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.0 MiB/s wr, 471 op/s; 182/1695 objects degraded (10.737%) 2024-04-14T13:47:06.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:05 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 182/1695 objects degraded (10.737%), 25 pgs degraded (PG_DEGRADED) 2024-04-14T13:47:06.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:05 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:06.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:05 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:06.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:05 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:47:06.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:05 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:47:06.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:05 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:06.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:05 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:06.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:05 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:47:06.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:05 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:47:06.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:05 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:47:06.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:05 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:47:06.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:05 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T13:47:06.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 285987 10.06 MB/sec execute 580 sec latency 73.678 ms 2024-04-14T13:47:06.583 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:06 smithi008 ceph-mon[67070]: pgmap v265: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 345 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.3 MiB/s wr, 517 op/s; 188/1746 objects degraded (10.767%) 2024-04-14T13:47:06.583 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:06 smithi008 ceph-mon[67070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T13:47:06.583 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:06 smithi008 ceph-mon[67070]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-14T13:47:07.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:06 smithi012 ceph-mon[53289]: pgmap v265: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 345 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.3 MiB/s wr, 517 op/s; 188/1746 objects degraded (10.767%) 2024-04-14T13:47:07.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:06 smithi012 ceph-mon[53289]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T13:47:07.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:06 smithi012 ceph-mon[53289]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-14T13:47:07.051 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:47:06 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:47:06.553+0000 7f3b1d9637c0 -1 osd.6 133 log_to_monitors true 2024-04-14T13:47:07.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 286495 10.05 MB/sec execute 581 sec latency 66.167 ms 2024-04-14T13:47:07.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:07 smithi008 ceph-mon[67070]: from='osd.6 [v2:172.21.15.12:6816/3931290909,v1:172.21.15.12:6817/3931290909]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T13:47:07.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:07 smithi008 ceph-mon[67070]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T13:47:08.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:07 smithi012 ceph-mon[53289]: from='osd.6 [v2:172.21.15.12:6816/3931290909,v1:172.21.15.12:6817/3931290909]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T13:47:08.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:07 smithi012 ceph-mon[53289]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T13:47:08.051 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:47:07 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:47:07.578+0000 7f3b0edbb700 -1 osd.6 133 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:47:08.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 286889 10.05 MB/sec execute 582 sec latency 78.419 ms 2024-04-14T13:47:08.647 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:08 smithi008 ceph-mon[67070]: pgmap v266: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.8 MiB/s wr, 471 op/s; 188/1746 objects degraded (10.767%) 2024-04-14T13:47:08.647 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:08 smithi008 ceph-mon[67070]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-14T13:47:08.647 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:08 smithi008 ceph-mon[67070]: from='osd.6 [v2:172.21.15.12:6816/3931290909,v1:172.21.15.12:6817/3931290909]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:47:08.647 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:08 smithi008 ceph-mon[67070]: osdmap e136: 8 total, 7 up, 8 in 2024-04-14T13:47:08.647 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:08 smithi008 ceph-mon[67070]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:47:09.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:08 smithi012 ceph-mon[53289]: pgmap v266: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.8 MiB/s wr, 471 op/s; 188/1746 objects degraded (10.767%) 2024-04-14T13:47:09.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:08 smithi012 ceph-mon[53289]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-14T13:47:09.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:08 smithi012 ceph-mon[53289]: from='osd.6 [v2:172.21.15.12:6816/3931290909,v1:172.21.15.12:6817/3931290909]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:47:09.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:08 smithi012 ceph-mon[53289]: osdmap e136: 8 total, 7 up, 8 in 2024-04-14T13:47:09.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:08 smithi012 ceph-mon[53289]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:47:09.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 287006 10.03 MB/sec execute 583 sec latency 777.922 ms 2024-04-14T13:47:09.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:09 smithi008 ceph-mon[67070]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T13:47:09.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:09 smithi008 ceph-mon[67070]: osd.6 [v2:172.21.15.12:6816/3931290909,v1:172.21.15.12:6817/3931290909] boot 2024-04-14T13:47:09.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:09 smithi008 ceph-mon[67070]: osdmap e137: 8 total, 8 up, 8 in 2024-04-14T13:47:09.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:09 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:47:09.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:09 smithi008 ceph-mon[67070]: osdmap e138: 8 total, 8 up, 8 in 2024-04-14T13:47:10.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:09 smithi012 ceph-mon[53289]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T13:47:10.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:09 smithi012 ceph-mon[53289]: osd.6 [v2:172.21.15.12:6816/3931290909,v1:172.21.15.12:6817/3931290909] boot 2024-04-14T13:47:10.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:09 smithi012 ceph-mon[53289]: osdmap e137: 8 total, 8 up, 8 in 2024-04-14T13:47:10.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:09 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:47:10.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:09 smithi012 ceph-mon[53289]: osdmap e138: 8 total, 8 up, 8 in 2024-04-14T13:47:10.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 287006 10.02 MB/sec execute 584 sec latency 1777.966 ms 2024-04-14T13:47:10.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:10 smithi008 ceph-mon[67070]: pgmap v269: 97 pgs: 1 unknown, 9 peering, 8 active+undersized, 17 active+undersized+degraded, 62 active+clean; 336 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 457 op/s; 122/1713 objects degraded (7.122%) 2024-04-14T13:47:10.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:10 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 122/1713 objects degraded (7.122%), 17 pgs degraded (PG_DEGRADED) 2024-04-14T13:47:10.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:10 smithi008 ceph-mon[67070]: osdmap e139: 8 total, 8 up, 8 in 2024-04-14T13:47:11.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:10 smithi012 ceph-mon[53289]: pgmap v269: 97 pgs: 1 unknown, 9 peering, 8 active+undersized, 17 active+undersized+degraded, 62 active+clean; 336 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 457 op/s; 122/1713 objects degraded (7.122%) 2024-04-14T13:47:11.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:10 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 122/1713 objects degraded (7.122%), 17 pgs degraded (PG_DEGRADED) 2024-04-14T13:47:11.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:10 smithi012 ceph-mon[53289]: osdmap e139: 8 total, 8 up, 8 in 2024-04-14T13:47:11.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 287148 10.01 MB/sec execute 585 sec latency 2470.012 ms 2024-04-14T13:47:12.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 287495 10.02 MB/sec execute 586 sec latency 666.194 ms 2024-04-14T13:47:12.619 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:12 smithi008 ceph-mon[67070]: pgmap v272: 97 pgs: 2 remapped+peering, 1 unknown, 9 peering, 3 active+undersized, 13 active+undersized+degraded, 69 active+clean; 332 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 3.8 MiB/s wr, 363 op/s; 86/1617 objects degraded (5.318%); 4.1 KiB/s, 5 objects/s recovering 2024-04-14T13:47:12.619 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:12 smithi008 ceph-mon[67070]: osdmap e140: 8 total, 8 up, 8 in 2024-04-14T13:47:12.619 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:12 smithi008 ceph-mon[67070]: osdmap e141: 8 total, 8 up, 8 in 2024-04-14T13:47:13.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:12 smithi012 ceph-mon[53289]: pgmap v272: 97 pgs: 2 remapped+peering, 1 unknown, 9 peering, 3 active+undersized, 13 active+undersized+degraded, 69 active+clean; 332 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 3.8 MiB/s wr, 363 op/s; 86/1617 objects degraded (5.318%); 4.1 KiB/s, 5 objects/s recovering 2024-04-14T13:47:13.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:12 smithi012 ceph-mon[53289]: osdmap e140: 8 total, 8 up, 8 in 2024-04-14T13:47:13.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:12 smithi012 ceph-mon[53289]: osdmap e141: 8 total, 8 up, 8 in 2024-04-14T13:47:13.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 287928 10.03 MB/sec execute 587 sec latency 645.843 ms 2024-04-14T13:47:14.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 288377 10.03 MB/sec execute 588 sec latency 988.251 ms 2024-04-14T13:47:14.396 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:14 smithi008 ceph-mon[67070]: osdmap e142: 8 total, 8 up, 8 in 2024-04-14T13:47:14.397 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:14 smithi008 ceph-mon[67070]: pgmap v276: 97 pgs: 1 unknown, 9 peering, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 84 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.4 MiB/s wr, 456 op/s; 49/1581 objects degraded (3.099%); 3.2 MiB/s, 60 keys/s, 23 objects/s recovering 2024-04-14T13:47:14.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:14 smithi012 ceph-mon[53289]: osdmap e142: 8 total, 8 up, 8 in 2024-04-14T13:47:14.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:14 smithi012 ceph-mon[53289]: pgmap v276: 97 pgs: 1 unknown, 9 peering, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 84 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.4 MiB/s wr, 456 op/s; 49/1581 objects degraded (3.099%); 3.2 MiB/s, 60 keys/s, 23 objects/s recovering 2024-04-14T13:47:15.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 288804 10.04 MB/sec execute 589 sec latency 67.190 ms 2024-04-14T13:47:15.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:15 smithi008 ceph-mon[67070]: osdmap e143: 8 total, 8 up, 8 in 2024-04-14T13:47:15.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:15 smithi012 ceph-mon[53289]: osdmap e143: 8 total, 8 up, 8 in 2024-04-14T13:47:16.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 289130 10.03 MB/sec execute 590 sec latency 60.286 ms 2024-04-14T13:47:16.417 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:16 smithi008 ceph-mon[67070]: pgmap v278: 97 pgs: 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 93 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6.3 MiB/s wr, 610 op/s; 49/1692 objects degraded (2.896%); 4.3 MiB/s, 56 keys/s, 35 objects/s recovering 2024-04-14T13:47:16.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:16 smithi012 ceph-mon[53289]: pgmap v278: 97 pgs: 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 93 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6.3 MiB/s wr, 610 op/s; 49/1692 objects degraded (2.896%); 4.3 MiB/s, 56 keys/s, 35 objects/s recovering 2024-04-14T13:47:17.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 289398 10.02 MB/sec execute 591 sec latency 74.795 ms 2024-04-14T13:47:17.430 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:17 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 49/1692 objects degraded (2.896%), 3 pgs degraded (PG_DEGRADED) 2024-04-14T13:47:17.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:17 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 49/1692 objects degraded (2.896%), 3 pgs degraded (PG_DEGRADED) 2024-04-14T13:47:18.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 289658 10.01 MB/sec execute 592 sec latency 48.588 ms 2024-04-14T13:47:18.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:18 smithi008 ceph-mon[67070]: pgmap v279: 97 pgs: 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 4.6 MiB/s wr, 441 op/s; 43/1689 objects degraded (2.546%); 3.0 MiB/s, 39 keys/s, 27 objects/s recovering 2024-04-14T13:47:18.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:18 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:18.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:18 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:47:18.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:18 smithi012 ceph-mon[53289]: pgmap v279: 97 pgs: 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 4.6 MiB/s wr, 441 op/s; 43/1689 objects degraded (2.546%); 3.0 MiB/s, 39 keys/s, 27 objects/s recovering 2024-04-14T13:47:18.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:18 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:18.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:18 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:47:19.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 290035 10.02 MB/sec execute 593 sec latency 38.653 ms 2024-04-14T13:47:19.462 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:19 smithi008 ceph-mon[67070]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43/1689 objects degraded (2.546%), 2 pgs degraded) 2024-04-14T13:47:19.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:19 smithi012 ceph-mon[53289]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43/1689 objects degraded (2.546%), 2 pgs degraded) 2024-04-14T13:47:20.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 290607 10.03 MB/sec execute 594 sec latency 42.739 ms 2024-04-14T13:47:20.470 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:20 smithi008 ceph-mon[67070]: pgmap v280: 97 pgs: 97 active+clean; 336 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 6.1 MiB/s wr, 675 op/s; 2.5 MiB/s, 27 objects/s recovering 2024-04-14T13:47:20.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:20 smithi012 ceph-mon[53289]: pgmap v280: 97 pgs: 97 active+clean; 336 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 6.1 MiB/s wr, 675 op/s; 2.5 MiB/s, 27 objects/s recovering 2024-04-14T13:47:21.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 290936 10.02 MB/sec execute 595 sec latency 52.898 ms 2024-04-14T13:47:21.495 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:21 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T13:47:21.496 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:21 smithi008 ceph-mon[67070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T13:47:21.496 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:21 smithi008 ceph-mon[67070]: Upgrade: osd.7 is safe to restart 2024-04-14T13:47:21.545 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:21 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T13:47:21.545 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:21 smithi012 ceph-mon[53289]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T13:47:21.546 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:21 smithi012 ceph-mon[53289]: Upgrade: osd.7 is safe to restart 2024-04-14T13:47:22.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 291234 10.03 MB/sec execute 596 sec latency 46.505 ms 2024-04-14T13:47:22.553 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:22 smithi008 ceph-mon[67070]: pgmap v281: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 5.1 MiB/s wr, 465 op/s; 664 KiB/s, 16 objects/s recovering 2024-04-14T13:47:22.702 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:22 smithi012 ceph-mon[53289]: pgmap v281: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 5.1 MiB/s wr, 465 op/s; 664 KiB/s, 16 objects/s recovering 2024-04-14T13:47:23.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 291490 10.02 MB/sec execute 597 sec latency 47.308 ms 2024-04-14T13:47:23.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:23 smithi008 ceph-mon[67070]: Upgrade: Updating osd.7 2024-04-14T13:47:23.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:23 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:23.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:23 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T13:47:23.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:23 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:47:23.601 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:23 smithi008 ceph-mon[67070]: Deploying daemon osd.7 on smithi012 2024-04-14T13:47:23.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:23 smithi012 ceph-mon[53289]: Upgrade: Updating osd.7 2024-04-14T13:47:23.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:23 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:23.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:23 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T13:47:23.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:23 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:47:23.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:23 smithi012 ceph-mon[53289]: Deploying daemon osd.7 on smithi012 2024-04-14T13:47:24.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 291769 10.02 MB/sec execute 598 sec latency 82.538 ms 2024-04-14T13:47:24.621 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:24 smithi012 ceph-mon[53289]: pgmap v282: 97 pgs: 97 active+clean; 338 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.2 MiB/s wr, 574 op/s; 536 KiB/s, 13 objects/s recovering 2024-04-14T13:47:24.632 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:24 smithi008 ceph-mon[67070]: pgmap v282: 97 pgs: 97 active+clean; 338 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.2 MiB/s wr, 574 op/s; 536 KiB/s, 13 objects/s recovering 2024-04-14T13:47:25.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 291964 10.01 MB/sec execute 599 sec latency 82.012 ms 2024-04-14T13:47:26.301 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:26 smithi012 systemd[1]: Stopping Ceph osd.7 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:47:26.385 INFO:teuthology.orchestra.run.smithi008.stdout: 5 cleanup 600 sec 2024-04-14T13:47:26.662 INFO:teuthology.orchestra.run.smithi008.stdout: 0 cleanup 600 sec 2024-04-14T13:47:26.663 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:47:26.663 INFO:teuthology.orchestra.run.smithi008.stdout: Operation Count AvgLat MaxLat 2024-04-14T13:47:26.663 INFO:teuthology.orchestra.run.smithi008.stdout: ---------------------------------------- 2024-04-14T13:47:26.663 INFO:teuthology.orchestra.run.smithi008.stdout: NTCreateX 192022 4.480 2410.936 2024-04-14T13:47:26.663 INFO:teuthology.orchestra.run.smithi008.stdout: Close 140924 0.143 13.174 2024-04-14T13:47:26.663 INFO:teuthology.orchestra.run.smithi008.stdout: Rename 8157 9.454 144.644 2024-04-14T13:47:26.663 INFO:teuthology.orchestra.run.smithi008.stdout: Unlink 38878 6.772 2396.387 2024-04-14T13:47:26.663 INFO:teuthology.orchestra.run.smithi008.stdout: Qpathinfo 174070 1.164 95.637 2024-04-14T13:47:26.663 INFO:teuthology.orchestra.run.smithi008.stdout: Qfileinfo 30374 0.008 3.714 2024-04-14T13:47:26.663 INFO:teuthology.orchestra.run.smithi008.stdout: Qfsinfo 32047 0.646 71.742 2024-04-14T13:47:26.663 INFO:teuthology.orchestra.run.smithi008.stdout: Sfileinfo 15636 12.536 1038.124 2024-04-14T13:47:26.663 INFO:teuthology.orchestra.run.smithi008.stdout: Find 67387 1.036 14.850 2024-04-14T13:47:26.663 INFO:teuthology.orchestra.run.smithi008.stdout: WriteX 94963 13.509 2607.854 2024-04-14T13:47:26.664 INFO:teuthology.orchestra.run.smithi008.stdout: ReadX 302258 0.010 7.908 2024-04-14T13:47:26.664 INFO:teuthology.orchestra.run.smithi008.stdout: LockX 632 0.400 3.858 2024-04-14T13:47:26.664 INFO:teuthology.orchestra.run.smithi008.stdout: UnlockX 632 0.360 5.881 2024-04-14T13:47:26.664 INFO:teuthology.orchestra.run.smithi008.stdout: Flush 13474 0.002 0.043 2024-04-14T13:47:26.664 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:47:26.664 INFO:teuthology.orchestra.run.smithi008.stdout:Throughput 10.0129 MB/sec 5 clients 5 procs max_latency=2607.861 ms 2024-04-14T13:47:26.665 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'umount /mnt/foo' 2024-04-14T13:47:26.752 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done' 2024-04-14T13:47:26.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:26 smithi012 ceph-mon[53289]: pgmap v283: 97 pgs: 97 active+clean; 338 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.0 MiB/s wr, 488 op/s; 23 KiB/s, 5 objects/s recovering 2024-04-14T13:47:26.801 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:26 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[40426]: 2024-04-14T13:47:26.558+0000 7fbfdc2a8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:47:26.801 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:26 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[40426]: 2024-04-14T13:47:26.558+0000 7fbfdc2a8700 -1 osd.7 143 *** Got signal Terminated *** 2024-04-14T13:47:26.801 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:26 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[40426]: 2024-04-14T13:47:26.558+0000 7fbfdc2a8700 -1 osd.7 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T13:47:26.817 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:26 smithi008 ceph-mon[67070]: pgmap v283: 97 pgs: 97 active+clean; 338 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.0 MiB/s wr, 488 op/s; 23 KiB/s, 5 objects/s recovering 2024-04-14T13:47:26.886 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 5' 2024-04-14T13:47:26.955 INFO:teuthology.orchestra.run.smithi008.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-14T13:47:26.955 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:47:26.955 INFO:teuthology.orchestra.run.smithi008.stdout:Running for 5 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 1 secs 2024-04-14T13:47:26.956 INFO:teuthology.orchestra.run.smithi008.stdout:3 of 5 processes prepared for launch 0 sec 2024-04-14T13:47:27.056 INFO:teuthology.orchestra.run.smithi008.stdout:5 of 5 processes prepared for launch 0 sec 2024-04-14T13:47:27.056 INFO:teuthology.orchestra.run.smithi008.stdout:releasing clients 2024-04-14T13:47:27.624 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:27 smithi008 ceph-mon[67070]: osd.7 marked itself down and dead 2024-04-14T13:47:27.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:27 smithi012 ceph-mon[53289]: osd.7 marked itself down and dead 2024-04-14T13:47:28.495 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:28 smithi012 podman[64480]: 2024-04-14 13:47:28.223278479 +0000 UTC m=+1.792229301 container died 99bab39bb43479ea69e1351ef93f41c3fde0152a1b4eef49c26a7b2fab4860b8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7, io.openshift.tags=base centos centos-stream, vcs-type=git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, com.redhat.component=centos-stream-container, io.openshift.expose-services=, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-14T13:47:28.781 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:28 smithi008 ceph-mon[67070]: pgmap v284: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.3 MiB/s wr, 441 op/s; 21 KiB/s, 4 objects/s recovering 2024-04-14T13:47:28.781 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:28 smithi008 ceph-mon[67070]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T13:47:28.781 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:28 smithi008 ceph-mon[67070]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-14T13:47:28.781 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:28 smithi008 ceph-mon[67070]: osdmap e144: 8 total, 7 up, 8 in 2024-04-14T13:47:28.781 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:28 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:28.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:28 smithi012 ceph-mon[53289]: pgmap v284: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.3 MiB/s wr, 441 op/s; 21 KiB/s, 4 objects/s recovering 2024-04-14T13:47:28.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:28 smithi012 ceph-mon[53289]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T13:47:28.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:28 smithi012 ceph-mon[53289]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-14T13:47:28.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:28 smithi012 ceph-mon[53289]: osdmap e144: 8 total, 7 up, 8 in 2024-04-14T13:47:28.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:28 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:28.801 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:28 smithi012 podman[64480]: 2024-04-14 13:47:28.55782287 +0000 UTC m=+2.126773688 container cleanup 99bab39bb43479ea69e1351ef93f41c3fde0152a1b4eef49c26a7b2fab4860b8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, architecture=x86_64, vendor=Red Hat, Inc., vcs-type=git, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=) 2024-04-14T13:47:28.801 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:28 smithi012 bash[64480]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7 2024-04-14T13:47:29.057 INFO:teuthology.orchestra.run.smithi008.stdout: 5 226 13.26 MB/sec execute 1 sec latency 1051.117 ms 2024-04-14T13:47:29.301 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:28 smithi012 podman[64498]: 2024-04-14 13:47:28.81347968 +0000 UTC m=+0.586565545 container remove 99bab39bb43479ea69e1351ef93f41c3fde0152a1b4eef49c26a7b2fab4860b8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, GIT_CLEAN=True, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, ceph=True, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux ) 2024-04-14T13:47:29.596 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:29 smithi008 ceph-mon[67070]: osdmap e145: 8 total, 7 up, 8 in 2024-04-14T13:47:29.713 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:29 smithi012 ceph-mon[53289]: osdmap e145: 8 total, 7 up, 8 in 2024-04-14T13:47:29.714 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:29 smithi012 podman[64580]: 2024-04-14 13:47:29.233484718 +0000 UTC m=+0.022642498 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:47:30.033 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:29 smithi012 podman[64580]: 2024-04-14 13:47:29.710973617 +0000 UTC m=+0.500131386 container create 14f70a0d6934edfbcc2c76e17cba40071652daa091205430b3e7403c6933a9a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-14T13:47:30.033 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:30 smithi012 podman[64580]: 2024-04-14 13:47:30.030813302 +0000 UTC m=+0.819971070 container init 14f70a0d6934edfbcc2c76e17cba40071652daa091205430b3e7403c6933a9a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-deactivate, RELEASE=main-a971fd5, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-14T13:47:30.056 INFO:teuthology.orchestra.run.smithi008.stdout: 5 380 18.60 MB/sec execute 2 sec latency 44.396 ms 2024-04-14T13:47:30.300 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:30 smithi012 podman[64580]: 2024-04-14 13:47:30.034637642 +0000 UTC m=+0.823795420 container start 14f70a0d6934edfbcc2c76e17cba40071652daa091205430b3e7403c6933a9a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-14T13:47:30.301 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:30 smithi012 podman[64580]: 2024-04-14 13:47:30.131086038 +0000 UTC m=+0.920243818 container attach 14f70a0d6934edfbcc2c76e17cba40071652daa091205430b3e7403c6933a9a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-14T13:47:30.641 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:30 smithi008 ceph-mon[67070]: pgmap v287: 97 pgs: 2 active+undersized, 6 active+undersized+degraded, 13 peering, 9 stale+active+clean, 67 active+clean; 353 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.5 MiB/s wr, 489 op/s; 45/1695 objects degraded (2.655%) 2024-04-14T13:47:30.641 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:30 smithi008 ceph-mon[67070]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-14T13:47:30.641 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:30 smithi008 ceph-mon[67070]: Health check failed: Degraded data redundancy: 45/1695 objects degraded (2.655%), 6 pgs degraded (PG_DEGRADED) 2024-04-14T13:47:30.717 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:30 smithi012 ceph-mon[53289]: pgmap v287: 97 pgs: 2 active+undersized, 6 active+undersized+degraded, 13 peering, 9 stale+active+clean, 67 active+clean; 353 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.5 MiB/s wr, 489 op/s; 45/1695 objects degraded (2.655%) 2024-04-14T13:47:30.717 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:30 smithi012 ceph-mon[53289]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-14T13:47:30.717 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:30 smithi012 ceph-mon[53289]: Health check failed: Degraded data redundancy: 45/1695 objects degraded (2.655%), 6 pgs degraded (PG_DEGRADED) 2024-04-14T13:47:30.717 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:30 smithi012 podman[64580]: 2024-04-14 13:47:30.391799361 +0000 UTC m=+1.180957130 container died 14f70a0d6934edfbcc2c76e17cba40071652daa091205430b3e7403c6933a9a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5) 2024-04-14T13:47:31.056 INFO:teuthology.orchestra.run.smithi008.stdout: 5 579 22.32 MB/sec execute 3 sec latency 132.735 ms 2024-04-14T13:47:31.301 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:31 smithi012 podman[64606]: 2024-04-14 13:47:31.011504781 +0000 UTC m=+0.613952691 container remove 14f70a0d6934edfbcc2c76e17cba40071652daa091205430b3e7403c6933a9a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:47:31.301 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:31 smithi012 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.7.service: Deactivated successfully. 2024-04-14T13:47:31.301 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:31 smithi012 systemd[1]: Stopped Ceph osd.7 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:47:31.301 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:31 smithi012 systemd[1]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.7.service: Consumed 8min 16.278s CPU time. 2024-04-14T13:47:31.301 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:31 smithi012 systemd[1]: Starting Ceph osd.7 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:47:31.777 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:31 smithi012 podman[64708]: 2024-04-14 13:47:31.382246475 +0000 UTC m=+0.018913597 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:47:31.778 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:31 smithi012 podman[64708]: 2024-04-14 13:47:31.504025836 +0000 UTC m=+0.140692955 container create 836abf90298e0f126a32a80ace296636cb0922f5818b0480791e642fd6cc81fb (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-activate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:47:31.778 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:31 smithi012 podman[64708]: 2024-04-14 13:47:31.774413865 +0000 UTC m=+0.411080998 container init 836abf90298e0f126a32a80ace296636cb0922f5818b0480791e642fd6cc81fb (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-activate, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-14T13:47:32.050 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:31 smithi012 podman[64708]: 2024-04-14 13:47:31.778200924 +0000 UTC m=+0.414868049 container start 836abf90298e0f126a32a80ace296636cb0922f5818b0480791e642fd6cc81fb (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240403) 2024-04-14T13:47:32.051 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:31 smithi012 podman[64708]: 2024-04-14 13:47:31.8825754 +0000 UTC m=+0.519242530 container attach 836abf90298e0f126a32a80ace296636cb0922f5818b0480791e642fd6cc81fb (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-activate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS) 2024-04-14T13:47:32.056 INFO:teuthology.orchestra.run.smithi008.stdout: 5 899 19.51 MB/sec execute 4 sec latency 37.343 ms 2024-04-14T13:47:32.551 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:32 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-activate[64723]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-14T13:47:32.551 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:32 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-activate[64723]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-14T13:47:32.551 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:32 smithi012 bash[64708]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-14T13:47:32.551 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:32 smithi012 bash[64708]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-14T13:47:32.551 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:32 smithi012 bash[64708]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-14T13:47:32.551 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:32 smithi012 bash[64708]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-14T13:47:32.551 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:32 smithi012 bash[64708]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-14T13:47:32.551 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:32 smithi012 bash[64708]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-14T13:47:32.551 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:32 smithi012 bash[64708]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-14T13:47:32.551 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:32 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-activate[64723]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-14T13:47:32.551 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:32 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-activate[64723]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-14T13:47:32.551 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:32 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-activate[64723]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-14T13:47:32.552 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:32 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-activate[64723]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-14T13:47:32.552 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:32 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-activate[64723]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-14T13:47:32.552 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:32 smithi012 podman[64708]: 2024-04-14 13:47:32.307604144 +0000 UTC m=+0.944271263 container died 836abf90298e0f126a32a80ace296636cb0922f5818b0480791e642fd6cc81fb (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403) 2024-04-14T13:47:32.682 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:32 smithi008 ceph-mon[67070]: pgmap v288: 97 pgs: 5 active+undersized, 20 active+undersized+degraded, 13 peering, 1 stale+active+clean, 58 active+clean; 326 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.6 MiB/s wr, 296 op/s; 115/1335 objects degraded (8.614%) 2024-04-14T13:47:32.948 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:32 smithi012 ceph-mon[53289]: pgmap v288: 97 pgs: 5 active+undersized, 20 active+undersized+degraded, 13 peering, 1 stale+active+clean, 58 active+clean; 326 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.6 MiB/s wr, 296 op/s; 115/1335 objects degraded (8.614%) 2024-04-14T13:47:33.056 INFO:teuthology.orchestra.run.smithi008.stdout: 5 cleanup 5 sec 2024-04-14T13:47:33.262 INFO:teuthology.orchestra.run.smithi008.stdout: 0 cleanup 5 sec 2024-04-14T13:47:33.263 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:47:33.263 INFO:teuthology.orchestra.run.smithi008.stdout: Operation Count AvgLat MaxLat 2024-04-14T13:47:33.263 INFO:teuthology.orchestra.run.smithi008.stdout: ---------------------------------------- 2024-04-14T13:47:33.263 INFO:teuthology.orchestra.run.smithi008.stdout: NTCreateX 806 7.787 50.035 2024-04-14T13:47:33.263 INFO:teuthology.orchestra.run.smithi008.stdout: Close 746 8.784 1051.105 2024-04-14T13:47:33.263 INFO:teuthology.orchestra.run.smithi008.stdout: Rename 45 11.780 37.337 2024-04-14T13:47:33.263 INFO:teuthology.orchestra.run.smithi008.stdout: Unlink 76 3.149 15.315 2024-04-14T13:47:33.263 INFO:teuthology.orchestra.run.smithi008.stdout: Qpathinfo 427 1.262 18.774 2024-04-14T13:47:33.263 INFO:teuthology.orchestra.run.smithi008.stdout: Qfileinfo 386 24.369 1007.732 2024-04-14T13:47:33.263 INFO:teuthology.orchestra.run.smithi008.stdout: Qfsinfo 152 0.431 2.597 2024-04-14T13:47:33.263 INFO:teuthology.orchestra.run.smithi008.stdout: Sfileinfo 151 12.234 22.340 2024-04-14T13:47:33.263 INFO:teuthology.orchestra.run.smithi008.stdout: Find 205 0.784 1.908 2024-04-14T13:47:33.263 INFO:teuthology.orchestra.run.smithi008.stdout: WriteX 1842 0.026 0.067 2024-04-14T13:47:33.263 INFO:teuthology.orchestra.run.smithi008.stdout: ReadX 825 0.021 0.347 2024-04-14T13:47:33.263 INFO:teuthology.orchestra.run.smithi008.stdout: Flush 120 12.881 20.605 2024-04-14T13:47:33.264 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:47:33.264 INFO:teuthology.orchestra.run.smithi008.stdout:Throughput 19.5116 MB/sec 5 clients 5 procs max_latency=1051.117 ms 2024-04-14T13:47:33.265 DEBUG:teuthology.parallel:result is None 2024-04-14T13:47:33.291 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:32 smithi012 podman[64839]: 2024-04-14 13:47:32.946217826 +0000 UTC m=+0.632861606 container remove 836abf90298e0f126a32a80ace296636cb0922f5818b0480791e642fd6cc81fb (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-activate, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-04-14T13:47:33.291 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:33 smithi012 podman[64901]: 2024-04-14 13:47:33.068028567 +0000 UTC m=+0.018969016 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:47:33.291 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:33 smithi012 podman[64901]: 2024-04-14 13:47:33.172447869 +0000 UTC m=+0.123388351 container create 7e4e731dec316eda365169f1085b788085bedd2dbd95fb07eee3be49bc11843c (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-14T13:47:33.551 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:33 smithi012 podman[64901]: 2024-04-14 13:47:33.391665711 +0000 UTC m=+0.342606153 container init 7e4e731dec316eda365169f1085b788085bedd2dbd95fb07eee3be49bc11843c (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-14T13:47:33.551 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:33 smithi012 podman[64901]: 2024-04-14 13:47:33.394670464 +0000 UTC m=+0.345610906 container start 7e4e731dec316eda365169f1085b788085bedd2dbd95fb07eee3be49bc11843c (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-04-14T13:47:33.551 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:33 smithi012 ceph-osd[64920]: -- 172.21.15.12:0/3177330968 <== mon.1 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5573d7f5d680 con 0x5573d7f26c00 2024-04-14T13:47:33.551 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:33 smithi012 bash[64901]: 7e4e731dec316eda365169f1085b788085bedd2dbd95fb07eee3be49bc11843c 2024-04-14T13:47:33.551 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:33 smithi012 systemd[1]: Started Ceph osd.7 for a8b03f6a-fa62-11ee-bc8e-c7b262605968. 2024-04-14T13:47:34.301 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:33 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:47:33.942+0000 7faecdfe67c0 -1 Falling back to public interface 2024-04-14T13:47:34.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:34 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:34.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:34 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:47:34.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:34 smithi012 ceph-mon[53289]: pgmap v289: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 320 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s wr, 338 op/s; 175/1230 objects degraded (14.228%) 2024-04-14T13:47:34.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:34 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:34.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:34 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:34.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:34 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:47:34.552 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:34 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:34.552 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:34 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:47:34.552 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:34 smithi008 ceph-mon[67070]: pgmap v289: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 320 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s wr, 338 op/s; 175/1230 objects degraded (14.228%) 2024-04-14T13:47:34.552 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:34 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:34.552 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:34 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:34.552 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:34 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:47:34.553 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:47:35.233 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:47:35.233 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (9m) 2m ago 18m 15.3M - 0.23.0 ba2b418f427c cfddb9a7f858 2024-04-14T13:47:35.233 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (5m) 2m ago 18m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 1e69eabdcc36 2024-04-14T13:47:35.233 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (5m) 35s ago 17m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 60c7a2992eb0 2024-04-14T13:47:35.233 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 2m ago 18m - - 2024-04-14T13:47:35.233 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (12m) 2m ago 12m 39.0M - 17.2.0 e1d6a67b021e 0550519f0a41 2024-04-14T13:47:35.233 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (12m) 35s ago 12m 15.2M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:47:35.233 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (8m) 2m ago 19m 508M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:47:35.234 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (7m) 35s ago 17m 453M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:47:35.234 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (6m) 2m ago 20m 49.3M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 673dd6f4c8d8 2024-04-14T13:47:35.234 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (6m) 35s ago 17m 44.4M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 d498dec29c4f 2024-04-14T13:47:35.234 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (12m) 2m ago 12m 284M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:47:35.234 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (12m) 35s ago 12m 91.0M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:47:35.234 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (9m) 2m ago 18m 11.4M - 1.3.1 1dbe0e931976 646dfca70be4 2024-04-14T13:47:35.234 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (8m) 35s ago 17m 19.3M - 1.3.1 1dbe0e931976 0a60dcd4044c 2024-04-14T13:47:35.234 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (4m) 2m ago 17m 326M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 29997ccc66ac 2024-04-14T13:47:35.234 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (4m) 2m ago 16m 272M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 97df88f10bd6 2024-04-14T13:47:35.234 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (3m) 2m ago 16m 205M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 9325b527c16c 2024-04-14T13:47:35.234 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (2m) 2m ago 15m 14.7M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 6868b5076d61 2024-04-14T13:47:35.234 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (117s) 35s ago 15m 319M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 178f9462f5ba 2024-04-14T13:47:35.234 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (79s) 35s ago 14m 262M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 25e416621d40 2024-04-14T13:47:35.234 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (40s) 35s ago 14m 11.5M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 97f0b691cf87 2024-04-14T13:47:35.234 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 starting - - - 2705M 2024-04-14T13:47:35.235 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (7m) 2m ago 17m 69.9M - 2.33.4 514e6a882f6e 12f97f75d620 2024-04-14T13:47:35.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:35 smithi012 ceph-mon[53289]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-14T13:47:35.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:35 smithi012 ceph-mon[53289]: from='client.34442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:47:35.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:35 smithi012 ceph-mon[53289]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:47:35.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:35 smithi008 ceph-mon[67070]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-14T13:47:35.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:35 smithi008 ceph-mon[67070]: from='client.34442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:47:35.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:35 smithi008 ceph-mon[67070]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:47:35.747 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:47:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:47:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:47:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:47:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:47:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:47:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:47:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:47:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 7 2024-04-14T13:47:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:47:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:47:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T13:47:35.748 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:47:35.748 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:47:35.748 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-14T13:47:35.748 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 11 2024-04-14T13:47:35.748 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:47:35.748 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:47:36.096 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:47:36.096 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:47:36.096 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:47:36.096 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:47:36.096 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-14T13:47:36.096 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-04-14T13:47:36.097 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-14T13:47:36.097 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-04-14T13:47:36.097 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:47:36.097 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "13/23 daemons upgraded", 2024-04-14T13:47:36.097 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T13:47:36.097 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:47:36.097 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:47:36.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:36 smithi012 ceph-mon[53289]: from='client.34450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:47:36.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:36 smithi012 ceph-mon[53289]: pgmap v290: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 332 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 16 MiB/s wr, 549 op/s; 204/1413 objects degraded (14.437%) 2024-04-14T13:47:36.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:36 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/336881074' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:47:36.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:36 smithi008 ceph-mon[67070]: from='client.34450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:47:36.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:36 smithi008 ceph-mon[67070]: pgmap v290: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 332 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 16 MiB/s wr, 549 op/s; 204/1413 objects degraded (14.437%) 2024-04-14T13:47:36.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:36 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/336881074' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:47:36.513 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 204/1413 objects degraded (14.437%), 35 pgs degraded 2024-04-14T13:47:36.513 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:47:36.513 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-14T13:47:36.513 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-14T13:47:36.513 INFO:teuthology.orchestra.run.smithi008.stdout: osd.7 (root=default,host=smithi012) is down 2024-04-14T13:47:36.513 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-04-14T13:47:36.513 INFO:teuthology.orchestra.run.smithi008.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-04-14T13:47:36.513 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 204/1413 objects degraded (14.437%), 35 pgs degraded 2024-04-14T13:47:36.513 INFO:teuthology.orchestra.run.smithi008.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-04-14T13:47:36.513 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-04-14T13:47:36.513 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-04-14T13:47:36.513 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.4 is active+undersized+degraded, acting [1,0] 2024-04-14T13:47:36.514 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.5 is active+undersized+degraded, acting [0,4] 2024-04-14T13:47:36.514 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.7 is active+undersized+degraded, acting [6,2] 2024-04-14T13:47:36.514 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.8 is active+undersized+degraded, acting [0,1] 2024-04-14T13:47:36.514 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-04-14T13:47:36.514 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-04-14T13:47:36.514 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.b is active+undersized+degraded, acting [4,5] 2024-04-14T13:47:36.514 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-04-14T13:47:36.514 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.12 is active+undersized+degraded, acting [5,3] 2024-04-14T13:47:36.514 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.18 is active+undersized+degraded, acting [5,4] 2024-04-14T13:47:36.514 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1a is active+undersized+degraded, acting [6,4] 2024-04-14T13:47:36.514 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1b is active+undersized+degraded, acting [3,6] 2024-04-14T13:47:36.514 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1c is active+undersized+degraded, acting [5,2] 2024-04-14T13:47:36.514 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1d is active+undersized+degraded, acting [6,0] 2024-04-14T13:47:36.514 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1f is active+undersized+degraded, acting [0,4] 2024-04-14T13:47:36.514 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-04-14T13:47:36.514 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.8 is active+undersized+degraded, acting [3,1] 2024-04-14T13:47:36.515 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.9 is active+undersized+degraded, acting [4,2] 2024-04-14T13:47:36.515 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.d is active+undersized+degraded, acting [5,6] 2024-04-14T13:47:36.515 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.e is active+undersized+degraded, acting [4,1] 2024-04-14T13:47:36.515 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2024-04-14T13:47:36.515 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.11 is active+undersized+degraded, acting [4,6] 2024-04-14T13:47:36.515 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.12 is active+undersized+degraded, acting [0,3] 2024-04-14T13:47:36.515 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.13 is active+undersized+degraded, acting [4,2] 2024-04-14T13:47:36.515 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.14 is active+undersized+degraded, acting [4,6] 2024-04-14T13:47:36.515 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-04-14T13:47:36.515 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.16 is active+undersized+degraded, acting [5,1] 2024-04-14T13:47:36.515 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1b is active+undersized+degraded, acting [0,4] 2024-04-14T13:47:36.515 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-04-14T13:47:36.515 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.5 is active+undersized+degraded, acting [6,4] 2024-04-14T13:47:36.515 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.a is active+undersized+degraded, acting [6,2] 2024-04-14T13:47:36.515 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-04-14T13:47:37.206 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:37 smithi012 ceph-mon[53289]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:47:37.206 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:37 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 204/1413 objects degraded (14.437%), 35 pgs degraded (PG_DEGRADED) 2024-04-14T13:47:37.206 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:37 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/3985324684' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:47:37.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:37 smithi008 ceph-mon[67070]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:47:37.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:37 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 204/1413 objects degraded (14.437%), 35 pgs degraded (PG_DEGRADED) 2024-04-14T13:47:37.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:37 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/3985324684' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:47:38.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:38 smithi012 ceph-mon[53289]: pgmap v291: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 319 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 13 MiB/s wr, 450 op/s; 203/1323 objects degraded (15.344%) 2024-04-14T13:47:38.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:38 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:38.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:38 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:38.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:38 smithi008 ceph-mon[67070]: pgmap v291: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 319 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 13 MiB/s wr, 450 op/s; 203/1323 objects degraded (15.344%) 2024-04-14T13:47:38.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:38 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:38.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:38 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:40.717 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:40 smithi012 ceph-mon[53289]: pgmap v292: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 287 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s wr, 307 op/s; 168/963 objects degraded (17.445%) 2024-04-14T13:47:40.717 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:40 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:40.718 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:40 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:40.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:40 smithi008 ceph-mon[67070]: pgmap v292: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 287 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s wr, 307 op/s; 168/963 objects degraded (17.445%) 2024-04-14T13:47:40.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:40 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:40.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:40 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:41.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:41 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 83/510 objects degraded (16.275%), 34 pgs degraded (PG_DEGRADED) 2024-04-14T13:47:41.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:41 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 83/510 objects degraded (16.275%), 34 pgs degraded (PG_DEGRADED) 2024-04-14T13:47:42.391 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:42 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:47:42.130+0000 7faecdfe67c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-04-14T13:47:42.733 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:42 smithi012 ceph-mon[53289]: pgmap v293: 97 pgs: 8 active+undersized, 34 active+undersized+degraded, 55 active+clean; 245 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.8 MiB/s wr, 284 op/s; 83/510 objects degraded (16.275%) 2024-04-14T13:47:42.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:42 smithi008 ceph-mon[67070]: pgmap v293: 97 pgs: 8 active+undersized, 34 active+undersized+degraded, 55 active+clean; 245 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.8 MiB/s wr, 284 op/s; 83/510 objects degraded (16.275%) 2024-04-14T13:47:44.050 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:43 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:47:43.659+0000 7faecdfe67c0 -1 osd.7 143 log_to_monitors true 2024-04-14T13:47:44.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:44.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:44.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:47:44.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:47:44.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: pgmap v294: 97 pgs: 8 active+undersized, 34 active+undersized+degraded, 55 active+clean; 235 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.6 MiB/s wr, 254 op/s; 80/456 objects degraded (17.544%) 2024-04-14T13:47:44.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: pgmap v295: 97 pgs: 8 active+undersized, 34 active+undersized+degraded, 55 active+clean; 235 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.6 MiB/s wr, 217 op/s; 80/456 objects degraded (17.544%) 2024-04-14T13:47:44.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:44.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:44.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:47:44.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:47:44.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:47:44.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:47:44.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:47:44.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: Upgrade: Setting container_image for all osd 2024-04-14T13:47:44.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:44.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-14T13:47:44.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-14T13:47:44.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-14T13:47:44.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-14T13:47:44.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-14T13:47:44.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-14T13:47:44.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-14T13:47:44.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-14T13:47:44.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-14T13:47:44.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-14T13:47:44.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-14T13:47:44.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-14T13:47:44.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-14T13:47:44.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-14T13:47:44.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-14T13:47:44.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-14T13:47:44.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: Upgrade: Setting require_osd_release to 19 squid 2024-04-14T13:47:44.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-14T13:47:44.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='osd.7 [v2:172.21.15.12:6824/3283676118,v1:172.21.15.12:6825/3283676118]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T13:47:44.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-mon[53289]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T13:47:44.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:44.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:44.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:47:44.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:47:44.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: pgmap v294: 97 pgs: 8 active+undersized, 34 active+undersized+degraded, 55 active+clean; 235 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.6 MiB/s wr, 254 op/s; 80/456 objects degraded (17.544%) 2024-04-14T13:47:44.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: pgmap v295: 97 pgs: 8 active+undersized, 34 active+undersized+degraded, 55 active+clean; 235 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.6 MiB/s wr, 217 op/s; 80/456 objects degraded (17.544%) 2024-04-14T13:47:44.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:44.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:44.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:47:44.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:47:44.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:47:44.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:47:44.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:47:44.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: Upgrade: Setting container_image for all osd 2024-04-14T13:47:44.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:44.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-14T13:47:44.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-14T13:47:44.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-14T13:47:44.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-14T13:47:44.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-14T13:47:44.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-14T13:47:44.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-14T13:47:44.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-14T13:47:44.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-14T13:47:44.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-14T13:47:44.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-14T13:47:44.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-14T13:47:44.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-14T13:47:44.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-14T13:47:44.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-14T13:47:44.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-14T13:47:44.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: Upgrade: Setting require_osd_release to 19 squid 2024-04-14T13:47:44.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-14T13:47:44.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='osd.7 [v2:172.21.15.12:6824/3283676118,v1:172.21.15.12:6825/3283676118]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T13:47:44.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:44 smithi008 ceph-mon[67070]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T13:47:45.051 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:47:44 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:47:44.726+0000 7faebb436700 -1 osd.7 143 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:47:45.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:45 smithi012 ceph-mon[53289]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-14T13:47:45.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:45 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-14T13:47:45.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:45 smithi012 ceph-mon[53289]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-14T13:47:45.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:45 smithi012 ceph-mon[53289]: osdmap e146: 8 total, 7 up, 8 in 2024-04-14T13:47:45.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:45 smithi012 ceph-mon[53289]: from='osd.7 [v2:172.21.15.12:6824/3283676118,v1:172.21.15.12:6825/3283676118]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:47:45.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:45 smithi012 ceph-mon[53289]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:47:45.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:45 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi008.wqbvfi"]}]: dispatch 2024-04-14T13:47:45.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:45 smithi012 ceph-mon[53289]: Upgrade: It appears safe to stop mds.foofs.smithi008.wqbvfi 2024-04-14T13:47:45.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:45 smithi012 ceph-mon[53289]: from='osd.7 ' entity='osd.7' 2024-04-14T13:47:45.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:45 smithi008 ceph-mon[67070]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-14T13:47:45.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:45 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-14T13:47:45.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:45 smithi008 ceph-mon[67070]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-14T13:47:45.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:45 smithi008 ceph-mon[67070]: osdmap e146: 8 total, 7 up, 8 in 2024-04-14T13:47:45.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:45 smithi008 ceph-mon[67070]: from='osd.7 [v2:172.21.15.12:6824/3283676118,v1:172.21.15.12:6825/3283676118]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:47:45.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:45 smithi008 ceph-mon[67070]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-14T13:47:45.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:45 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi008.wqbvfi"]}]: dispatch 2024-04-14T13:47:45.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:45 smithi008 ceph-mon[67070]: Upgrade: It appears safe to stop mds.foofs.smithi008.wqbvfi 2024-04-14T13:47:45.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:45 smithi008 ceph-mon[67070]: from='osd.7 ' entity='osd.7' 2024-04-14T13:47:46.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:46 smithi008 ceph-mon[67070]: pgmap v297: 97 pgs: 11 active+undersized, 31 active+undersized+degraded, 55 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 184 KiB/s wr, 47 op/s; 77/429 objects degraded (17.949%) 2024-04-14T13:47:46.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:46 smithi008 ceph-mon[67070]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T13:47:46.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:46 smithi008 ceph-mon[67070]: osd.7 [v2:172.21.15.12:6824/3283676118,v1:172.21.15.12:6825/3283676118] boot 2024-04-14T13:47:46.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:46 smithi008 ceph-mon[67070]: osdmap e147: 8 total, 8 up, 8 in 2024-04-14T13:47:46.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:46 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:47:46.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:46 smithi012 ceph-mon[53289]: pgmap v297: 97 pgs: 11 active+undersized, 31 active+undersized+degraded, 55 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 184 KiB/s wr, 47 op/s; 77/429 objects degraded (17.949%) 2024-04-14T13:47:46.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:46 smithi012 ceph-mon[53289]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T13:47:46.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:46 smithi012 ceph-mon[53289]: osd.7 [v2:172.21.15.12:6824/3283676118,v1:172.21.15.12:6825/3283676118] boot 2024-04-14T13:47:46.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:46 smithi012 ceph-mon[53289]: osdmap e147: 8 total, 8 up, 8 in 2024-04-14T13:47:46.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:46 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:47:47.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:47 smithi008 ceph-mon[67070]: Upgrade: Updating mds.foofs.smithi008.wqbvfi 2024-04-14T13:47:47.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:47 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:47.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:47 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.wqbvfi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-14T13:47:47.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:47 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:47:47.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:47 smithi008 ceph-mon[67070]: Deploying daemon mds.foofs.smithi008.wqbvfi on smithi008 2024-04-14T13:47:47.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:47 smithi008 ceph-mon[67070]: osdmap e148: 8 total, 8 up, 8 in 2024-04-14T13:47:47.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:47 smithi012 ceph-mon[53289]: Upgrade: Updating mds.foofs.smithi008.wqbvfi 2024-04-14T13:47:47.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:47 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:47.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:47 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.wqbvfi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-14T13:47:47.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:47 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:47:47.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:47 smithi012 ceph-mon[53289]: Deploying daemon mds.foofs.smithi008.wqbvfi on smithi008 2024-04-14T13:47:47.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:47 smithi012 ceph-mon[53289]: osdmap e148: 8 total, 8 up, 8 in 2024-04-14T13:47:48.339 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:48 smithi008 ceph-mon[67070]: pgmap v300: 97 pgs: 3 peering, 16 active+undersized, 23 active+undersized+degraded, 55 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 7.2 KiB/s wr, 3 op/s; 65/429 objects degraded (15.152%) 2024-04-14T13:47:48.340 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:48 smithi008 ceph-mon[67070]: Health check update: Degraded data redundancy: 77/429 objects degraded (17.949%), 31 pgs degraded (PG_DEGRADED) 2024-04-14T13:47:48.340 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:48 smithi008 ceph-mon[67070]: osdmap e149: 8 total, 8 up, 8 in 2024-04-14T13:47:48.340 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:48.340 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:47:48.551 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:47:48 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:47:48.356+0000 7f6e08501700 -1 bdev(0x55926739c000 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 3 2024-04-14T13:47:48.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:48 smithi012 ceph-mon[53289]: pgmap v300: 97 pgs: 3 peering, 16 active+undersized, 23 active+undersized+degraded, 55 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 7.2 KiB/s wr, 3 op/s; 65/429 objects degraded (15.152%) 2024-04-14T13:47:48.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:48 smithi012 ceph-mon[53289]: Health check update: Degraded data redundancy: 77/429 objects degraded (17.949%), 31 pgs degraded (PG_DEGRADED) 2024-04-14T13:47:48.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:48 smithi012 ceph-mon[53289]: osdmap e149: 8 total, 8 up, 8 in 2024-04-14T13:47:48.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:48.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:47:48.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:48 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008[67065]: 2024-04-14T13:47:48.337+0000 7fc99c555700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-14T13:47:48.603 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:47:48 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1[75673]: 2024-04-14T13:47:48.358+0000 7f3b6a798700 -1 bdev(0x55f3bcf2a000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 7 2024-04-14T13:47:48.603 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:47:48 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1[75673]: 2024-04-14T13:47:48.358+0000 7f3b64f8d700 -1 bdev(0x55f3bcf2a000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 8 2024-04-14T13:47:49.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:49 smithi012 ceph-mon[53289]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-14T13:47:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:49 smithi012 ceph-mon[53289]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-14T13:47:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:49 smithi012 ceph-mon[53289]: osdmap e150: 8 total, 8 up, 8 in 2024-04-14T13:47:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:49 smithi012 ceph-mon[53289]: Standby daemon mds.foofs.smithi012.vjovzp assigned to filesystem foofs as rank 0 2024-04-14T13:47:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:49 smithi012 ceph-mon[53289]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-14T13:47:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:49 smithi012 ceph-mon[53289]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-14T13:47:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:49 smithi012 ceph-mon[53289]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-14T13:47:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:49 smithi012 ceph-mon[53289]: fsmap foofs:1/1 {0=foofs.smithi012.vjovzp=up:replay} 2024-04-14T13:47:49.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wqbvfi"}]: dispatch 2024-04-14T13:47:49.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:49 smithi008 ceph-mon[67070]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-14T13:47:49.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:49 smithi008 ceph-mon[67070]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-14T13:47:49.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:49 smithi008 ceph-mon[67070]: osdmap e150: 8 total, 8 up, 8 in 2024-04-14T13:47:49.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:49 smithi008 ceph-mon[67070]: Standby daemon mds.foofs.smithi012.vjovzp assigned to filesystem foofs as rank 0 2024-04-14T13:47:49.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:49 smithi008 ceph-mon[67070]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-14T13:47:49.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:49 smithi008 ceph-mon[67070]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-14T13:47:49.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:49 smithi008 ceph-mon[67070]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-14T13:47:49.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:49 smithi008 ceph-mon[67070]: fsmap foofs:1/1 {0=foofs.smithi012.vjovzp=up:replay} 2024-04-14T13:47:49.604 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wqbvfi"}]: dispatch 2024-04-14T13:47:50.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:50 smithi008 ceph-mon[67070]: pgmap v303: 97 pgs: 4 peering, 11 active+undersized, 11 active+undersized+degraded, 71 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s wr, 0 op/s; 37/429 objects degraded (8.625%); 1024 KiB/s, 15 objects/s recovering 2024-04-14T13:47:50.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:50 smithi008 ceph-mon[67070]: osdmap e151: 8 total, 8 up, 8 in 2024-04-14T13:47:50.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:50 smithi012 ceph-mon[53289]: pgmap v303: 97 pgs: 4 peering, 11 active+undersized, 11 active+undersized+degraded, 71 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s wr, 0 op/s; 37/429 objects degraded (8.625%); 1024 KiB/s, 15 objects/s recovering 2024-04-14T13:47:50.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:50 smithi012 ceph-mon[53289]: osdmap e151: 8 total, 8 up, 8 in 2024-04-14T13:47:51.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:51 smithi012 ceph-mon[53289]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 37/429 objects degraded (8.625%), 11 pgs degraded) 2024-04-14T13:47:51.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:51 smithi012 ceph-mon[53289]: mds.? [v2:172.21.15.12:6832/1210020061,v1:172.21.15.12:6833/1210020061] up:reconnect 2024-04-14T13:47:51.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:51 smithi012 ceph-mon[53289]: fsmap foofs:1/1 {0=foofs.smithi012.vjovzp=up:reconnect} 2024-04-14T13:47:51.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:51 smithi008 ceph-mon[67070]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 37/429 objects degraded (8.625%), 11 pgs degraded) 2024-04-14T13:47:51.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:51 smithi008 ceph-mon[67070]: mds.? [v2:172.21.15.12:6832/1210020061,v1:172.21.15.12:6833/1210020061] up:reconnect 2024-04-14T13:47:51.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:51 smithi008 ceph-mon[67070]: fsmap foofs:1/1 {0=foofs.smithi012.vjovzp=up:reconnect} 2024-04-14T13:47:52.702 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:52 smithi012 ceph-mon[53289]: pgmap v305: 97 pgs: 4 peering, 93 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.5 MiB/s rd, 464 B/s wr, 2 op/s; 1.2 MiB/s, 69 objects/s recovering 2024-04-14T13:47:52.702 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:52 smithi012 ceph-mon[53289]: reconnect by client.24373 172.21.15.12:0/1442085824 after 0 2024-04-14T13:47:52.702 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:52 smithi012 ceph-mon[53289]: reconnect by client.14560 172.21.15.8:0/2332098306 after 0 2024-04-14T13:47:52.702 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:52 smithi012 ceph-mon[53289]: mds.? [v2:172.21.15.12:6832/1210020061,v1:172.21.15.12:6833/1210020061] up:rejoin 2024-04-14T13:47:52.702 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:52 smithi012 ceph-mon[53289]: fsmap foofs:1/1 {0=foofs.smithi012.vjovzp=up:rejoin} 2024-04-14T13:47:52.702 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:52 smithi012 ceph-mon[53289]: daemon mds.foofs.smithi012.vjovzp is now active in filesystem foofs as rank 0 2024-04-14T13:47:52.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:52 smithi008 ceph-mon[67070]: pgmap v305: 97 pgs: 4 peering, 93 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.5 MiB/s rd, 464 B/s wr, 2 op/s; 1.2 MiB/s, 69 objects/s recovering 2024-04-14T13:47:52.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:52 smithi008 ceph-mon[67070]: reconnect by client.24373 172.21.15.12:0/1442085824 after 0 2024-04-14T13:47:52.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:52 smithi008 ceph-mon[67070]: reconnect by client.14560 172.21.15.8:0/2332098306 after 0 2024-04-14T13:47:52.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:52 smithi008 ceph-mon[67070]: mds.? [v2:172.21.15.12:6832/1210020061,v1:172.21.15.12:6833/1210020061] up:rejoin 2024-04-14T13:47:52.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:52 smithi008 ceph-mon[67070]: fsmap foofs:1/1 {0=foofs.smithi012.vjovzp=up:rejoin} 2024-04-14T13:47:52.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:52 smithi008 ceph-mon[67070]: daemon mds.foofs.smithi012.vjovzp is now active in filesystem foofs as rank 0 2024-04-14T13:47:53.718 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:53 smithi012 ceph-mon[53289]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-14T13:47:53.718 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:53 smithi012 ceph-mon[53289]: mds.? [v2:172.21.15.12:6832/1210020061,v1:172.21.15.12:6833/1210020061] up:active 2024-04-14T13:47:53.718 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:53 smithi012 ceph-mon[53289]: fsmap foofs:1 {0=foofs.smithi012.vjovzp=up:active} 2024-04-14T13:47:53.767 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:53 smithi008 ceph-mon[67070]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-14T13:47:53.767 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:53 smithi008 ceph-mon[67070]: mds.? [v2:172.21.15.12:6832/1210020061,v1:172.21.15.12:6833/1210020061] up:active 2024-04-14T13:47:53.767 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:53 smithi008 ceph-mon[67070]: fsmap foofs:1 {0=foofs.smithi012.vjovzp=up:active} 2024-04-14T13:47:54.700 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:54 smithi008 ceph-mon[67070]: pgmap v306: 97 pgs: 1 peering, 96 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 MiB/s rd, 511 B/s wr, 9 op/s; 897 KiB/s, 50 objects/s recovering 2024-04-14T13:47:54.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:54 smithi012 ceph-mon[53289]: pgmap v306: 97 pgs: 1 peering, 96 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 MiB/s rd, 511 B/s wr, 9 op/s; 897 KiB/s, 50 objects/s recovering 2024-04-14T13:47:56.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:56 smithi008 ceph-mon[67070]: pgmap v307: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 29 MiB/s rd, 414 B/s wr, 11 op/s; 727 KiB/s, 37 objects/s recovering 2024-04-14T13:47:56.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:56 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:56.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:56 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:56.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:56 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:47:56.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:56 smithi008 ceph-mon[67070]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-14T13:47:56.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:56 smithi008 ceph-mon[67070]: mds.? [v2:172.21.15.8:6834/511465636,v1:172.21.15.8:6835/511465636] up:boot 2024-04-14T13:47:56.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:56 smithi008 ceph-mon[67070]: fsmap foofs:1 {0=foofs.smithi012.vjovzp=up:active} 1 up:standby 2024-04-14T13:47:56.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:56 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wqbvfi"}]: dispatch 2024-04-14T13:47:57.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:56 smithi012 ceph-mon[53289]: pgmap v307: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 29 MiB/s rd, 414 B/s wr, 11 op/s; 727 KiB/s, 37 objects/s recovering 2024-04-14T13:47:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:56 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:56 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:47:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:56 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:47:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:56 smithi012 ceph-mon[53289]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-14T13:47:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:56 smithi012 ceph-mon[53289]: mds.? [v2:172.21.15.8:6834/511465636,v1:172.21.15.8:6835/511465636] up:boot 2024-04-14T13:47:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:56 smithi012 ceph-mon[53289]: fsmap foofs:1 {0=foofs.smithi012.vjovzp=up:active} 1 up:standby 2024-04-14T13:47:57.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:56 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wqbvfi"}]: dispatch 2024-04-14T13:47:58.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:47:58 smithi008 ceph-mon[67070]: pgmap v308: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 MiB/s rd, 8.9 KiB/s wr, 13 op/s; 612 KiB/s, 31 objects/s recovering 2024-04-14T13:47:59.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:47:58 smithi012 ceph-mon[53289]: pgmap v308: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 MiB/s rd, 8.9 KiB/s wr, 13 op/s; 612 KiB/s, 31 objects/s recovering 2024-04-14T13:48:00.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:00 smithi008 ceph-mon[67070]: pgmap v309: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 7.8 KiB/s wr, 14 op/s; 129 KiB/s, 24 objects/s recovering 2024-04-14T13:48:00.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:00 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:00.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:00 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:01.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:00 smithi012 ceph-mon[53289]: pgmap v309: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 7.8 KiB/s wr, 14 op/s; 129 KiB/s, 24 objects/s recovering 2024-04-14T13:48:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:00 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:00 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:02.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:02 smithi008 ceph-mon[67070]: pgmap v310: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 7.9 KiB/s wr, 12 op/s; 109 KiB/s, 8 objects/s recovering 2024-04-14T13:48:02.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:02 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:02.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:02 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:03.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:02 smithi012 ceph-mon[53289]: pgmap v310: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 7.9 KiB/s wr, 12 op/s; 109 KiB/s, 8 objects/s recovering 2024-04-14T13:48:03.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:02 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:03.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:02 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:04.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:04 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:04.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:04 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:48:04.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:04 smithi008 ceph-mon[67070]: pgmap v311: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 7.6 KiB/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-04-14T13:48:04.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:04 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:04.388 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:04 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:04.388 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:04 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:48:04.388 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:04 smithi012 ceph-mon[53289]: pgmap v311: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 7.6 KiB/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-04-14T13:48:04.388 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:04 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:06.459 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:06 smithi008 ceph-mon[67070]: pgmap v312: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 7.5 KiB/s wr, 7 op/s 2024-04-14T13:48:06.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:06 smithi012 ceph-mon[53289]: pgmap v312: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 7.5 KiB/s wr, 7 op/s 2024-04-14T13:48:06.891 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:48:07.592 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:48:07.592 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (9m) 7s ago 19m 24.9M - 0.23.0 ba2b418f427c cfddb9a7f858 2024-04-14T13:48:07.592 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (6m) 7s ago 19m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 1e69eabdcc36 2024-04-14T13:48:07.592 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (5m) 29s ago 18m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 60c7a2992eb0 2024-04-14T13:48:07.592 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 7s ago 18m - - 2024-04-14T13:48:07.592 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (12s) 7s ago 13m 11.2M - 19.0.0-2778-ga971fd5f b96c509eef84 50aacbc1bc98 2024-04-14T13:48:07.592 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (13m) 29s ago 13m 15.4M - 17.2.0 e1d6a67b021e 0375c2efd30f 2024-04-14T13:48:07.592 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (8m) 7s ago 20m 512M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:48:07.592 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (7m) 29s ago 18m 453M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:48:07.592 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (7m) 7s ago 20m 59.5M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 673dd6f4c8d8 2024-04-14T13:48:07.593 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (7m) 29s ago 18m 45.6M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 d498dec29c4f 2024-04-14T13:48:07.593 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (12m) 7s ago 12m 264M - 4.0 e1d6a67b021e 40799f019261 2024-04-14T13:48:07.593 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (12m) 29s ago 12m 91.0M - 4.0 e1d6a67b021e 2eef858850bc 2024-04-14T13:48:07.593 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (9m) 7s ago 18m 17.4M - 1.3.1 1dbe0e931976 646dfca70be4 2024-04-14T13:48:07.593 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (9m) 29s ago 18m 20.0M - 1.3.1 1dbe0e931976 0a60dcd4044c 2024-04-14T13:48:07.593 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (5m) 7s ago 17m 392M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 29997ccc66ac 2024-04-14T13:48:07.593 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (4m) 7s ago 17m 361M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 97df88f10bd6 2024-04-14T13:48:07.593 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (3m) 7s ago 16m 324M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 9325b527c16c 2024-04-14T13:48:07.593 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (3m) 7s ago 16m 427M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 6868b5076d61 2024-04-14T13:48:07.593 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (2m) 29s ago 15m 362M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 178f9462f5ba 2024-04-14T13:48:07.593 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (111s) 29s ago 15m 312M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 25e416621d40 2024-04-14T13:48:07.593 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (72s) 29s ago 15m 290M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 97f0b691cf87 2024-04-14T13:48:07.593 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (34s) 29s ago 14m 12.6M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 7e4e731dec31 2024-04-14T13:48:07.593 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (7m) 7s ago 17m 70.5M - 2.33.4 514e6a882f6e 12f97f75d620 2024-04-14T13:48:07.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:07 smithi008 ceph-mon[67070]: Detected new or changed devices on smithi008 2024-04-14T13:48:07.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:07 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:07.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:07 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:07.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:07 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:48:07.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:07 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:48:07.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:07 smithi008 ceph-mon[67070]: pgmap v313: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 810 KiB/s rd, 7.7 KiB/s wr, 5 op/s 2024-04-14T13:48:07.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:07 smithi008 ceph-mon[67070]: pgmap v314: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-14T13:48:07.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:07 smithi008 ceph-mon[67070]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:48:07.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:07 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:07.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:07 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:07.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:07 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:48:07.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:07 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:07.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:07 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:07.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:07 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:07.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:07 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:07.854 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:07 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi012.vjovzp"]}]: dispatch 2024-04-14T13:48:07.989 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:07 smithi012 ceph-mon[53289]: Detected new or changed devices on smithi008 2024-04-14T13:48:07.989 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:07 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:07.989 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:07 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:07.989 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:07 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:48:07.989 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:07 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:48:07.989 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:07 smithi012 ceph-mon[53289]: pgmap v313: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 810 KiB/s rd, 7.7 KiB/s wr, 5 op/s 2024-04-14T13:48:07.990 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:07 smithi012 ceph-mon[53289]: pgmap v314: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-14T13:48:07.990 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:07 smithi012 ceph-mon[53289]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:48:07.990 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:07 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:07.990 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:07 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:07.990 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:07 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:48:07.990 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:07 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:07.990 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:07 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:07.990 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:07 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:07.990 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:07 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:07.990 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:07 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi012.vjovzp"]}]: dispatch 2024-04-14T13:48:08.015 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:48:08.015 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:48:08.016 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:48:08.016 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:48:08.016 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:48:08.016 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:48:08.016 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:48:08.016 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:48:08.016 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 8 2024-04-14T13:48:08.016 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:48:08.016 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:48:08.016 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-14T13:48:08.016 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T13:48:08.016 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:48:08.016 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:48:08.016 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-14T13:48:08.017 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 13 2024-04-14T13:48:08.017 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:48:08.017 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:48:08.394 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:48:08.394 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:48:08.394 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:48:08.394 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:48:08.394 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-14T13:48:08.394 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-04-14T13:48:08.394 INFO:teuthology.orchestra.run.smithi008.stdout: "osd", 2024-04-14T13:48:08.394 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-14T13:48:08.394 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-04-14T13:48:08.394 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:48:08.394 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "15/23 daemons upgraded", 2024-04-14T13:48:08.395 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading mds daemons", 2024-04-14T13:48:08.395 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:48:08.395 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:48:08.817 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T13:48:08.817 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:48:08.818 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-14T13:48:09.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:08 smithi012 ceph-mon[53289]: Upgrade: It appears safe to stop mds.foofs.smithi012.vjovzp 2024-04-14T13:48:09.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:08 smithi012 ceph-mon[53289]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:48:09.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:08 smithi012 ceph-mon[53289]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:48:09.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:08 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/3141653492' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:09.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:08 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:09.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:08 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi012.vjovzp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-14T13:48:09.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:08 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:48:09.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:08 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/207419201' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:48:09.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:08 smithi008 ceph-mon[67070]: Upgrade: It appears safe to stop mds.foofs.smithi012.vjovzp 2024-04-14T13:48:09.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:08 smithi008 ceph-mon[67070]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:48:09.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:08 smithi008 ceph-mon[67070]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:48:09.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:08 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/3141653492' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:09.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:08 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:09.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:08 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi012.vjovzp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-14T13:48:09.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:08 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:48:09.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:08 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/207419201' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:48:10.028 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:09 smithi012 ceph-mon[53289]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:48:10.028 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:09 smithi012 ceph-mon[53289]: Upgrade: Updating mds.foofs.smithi012.vjovzp 2024-04-14T13:48:10.028 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:09 smithi012 ceph-mon[53289]: Deploying daemon mds.foofs.smithi012.vjovzp on smithi012 2024-04-14T13:48:10.028 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:09 smithi012 ceph-mon[53289]: pgmap v315: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 1.5 KiB/s wr, 0 op/s 2024-04-14T13:48:10.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:09 smithi008 ceph-mon[67070]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:48:10.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:09 smithi008 ceph-mon[67070]: Upgrade: Updating mds.foofs.smithi012.vjovzp 2024-04-14T13:48:10.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:09 smithi008 ceph-mon[67070]: Deploying daemon mds.foofs.smithi012.vjovzp on smithi012 2024-04-14T13:48:10.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:09 smithi008 ceph-mon[67070]: pgmap v315: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 1.5 KiB/s wr, 0 op/s 2024-04-14T13:48:11.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:10 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008[67065]: 2024-04-14T13:48:10.962+0000 7fc99c555700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-14T13:48:12.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:11 smithi012 ceph-mon[53289]: pgmap v316: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 210 B/s wr, 0 op/s 2024-04-14T13:48:12.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:11 smithi012 ceph-mon[53289]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-14T13:48:12.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:11 smithi012 ceph-mon[53289]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-14T13:48:12.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:11 smithi012 ceph-mon[53289]: osdmap e152: 8 total, 8 up, 8 in 2024-04-14T13:48:12.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:11 smithi012 ceph-mon[53289]: Standby daemon mds.foofs.smithi008.wqbvfi assigned to filesystem foofs as rank 0 2024-04-14T13:48:12.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:11 smithi012 ceph-mon[53289]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-14T13:48:12.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:11 smithi012 ceph-mon[53289]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-14T13:48:12.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:11 smithi012 ceph-mon[53289]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-14T13:48:12.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:11 smithi012 ceph-mon[53289]: fsmap foofs:1/1 {0=foofs.smithi008.wqbvfi=up:replay} 2024-04-14T13:48:12.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:11 smithi008 ceph-mon[67070]: pgmap v316: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 210 B/s wr, 0 op/s 2024-04-14T13:48:12.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:11 smithi008 ceph-mon[67070]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-14T13:48:12.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:11 smithi008 ceph-mon[67070]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-14T13:48:12.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:11 smithi008 ceph-mon[67070]: osdmap e152: 8 total, 8 up, 8 in 2024-04-14T13:48:12.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:11 smithi008 ceph-mon[67070]: Standby daemon mds.foofs.smithi008.wqbvfi assigned to filesystem foofs as rank 0 2024-04-14T13:48:12.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:11 smithi008 ceph-mon[67070]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-14T13:48:12.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:11 smithi008 ceph-mon[67070]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-14T13:48:12.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:11 smithi008 ceph-mon[67070]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-14T13:48:12.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:11 smithi008 ceph-mon[67070]: fsmap foofs:1/1 {0=foofs.smithi008.wqbvfi=up:replay} 2024-04-14T13:48:14.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:13 smithi012 ceph-mon[53289]: pgmap v318: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 397 B/s wr, 4 op/s 2024-04-14T13:48:14.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:13 smithi012 ceph-mon[53289]: reconnect by client.14560 172.21.15.8:0/2332098306 after 0 2024-04-14T13:48:14.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:13 smithi012 ceph-mon[53289]: reconnect by client.24373 172.21.15.12:0/1442085824 after 0.000999991 2024-04-14T13:48:14.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:13 smithi012 ceph-mon[53289]: mds.? [v2:172.21.15.8:6834/511465636,v1:172.21.15.8:6835/511465636] up:reconnect 2024-04-14T13:48:14.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:13 smithi012 ceph-mon[53289]: fsmap foofs:1/1 {0=foofs.smithi008.wqbvfi=up:reconnect} 2024-04-14T13:48:14.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:13 smithi008 ceph-mon[67070]: pgmap v318: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 397 B/s wr, 4 op/s 2024-04-14T13:48:14.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:13 smithi008 ceph-mon[67070]: reconnect by client.14560 172.21.15.8:0/2332098306 after 0 2024-04-14T13:48:14.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:13 smithi008 ceph-mon[67070]: reconnect by client.24373 172.21.15.12:0/1442085824 after 0.000999991 2024-04-14T13:48:14.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:13 smithi008 ceph-mon[67070]: mds.? [v2:172.21.15.8:6834/511465636,v1:172.21.15.8:6835/511465636] up:reconnect 2024-04-14T13:48:14.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:13 smithi008 ceph-mon[67070]: fsmap foofs:1/1 {0=foofs.smithi008.wqbvfi=up:reconnect} 2024-04-14T13:48:15.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:14 smithi012 ceph-mon[53289]: mds.? [v2:172.21.15.8:6834/511465636,v1:172.21.15.8:6835/511465636] up:rejoin 2024-04-14T13:48:15.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:14 smithi012 ceph-mon[53289]: fsmap foofs:1/1 {0=foofs.smithi008.wqbvfi=up:rejoin} 2024-04-14T13:48:15.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:14 smithi012 ceph-mon[53289]: daemon mds.foofs.smithi008.wqbvfi is now active in filesystem foofs as rank 0 2024-04-14T13:48:15.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:14 smithi008 ceph-mon[67070]: mds.? [v2:172.21.15.8:6834/511465636,v1:172.21.15.8:6835/511465636] up:rejoin 2024-04-14T13:48:15.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:14 smithi008 ceph-mon[67070]: fsmap foofs:1/1 {0=foofs.smithi008.wqbvfi=up:rejoin} 2024-04-14T13:48:15.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:14 smithi008 ceph-mon[67070]: daemon mds.foofs.smithi008.wqbvfi is now active in filesystem foofs as rank 0 2024-04-14T13:48:16.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:15 smithi012 ceph-mon[53289]: pgmap v319: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 MiB/s rd, 383 B/s wr, 8 op/s 2024-04-14T13:48:16.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:15 smithi012 ceph-mon[53289]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-14T13:48:16.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:15 smithi012 ceph-mon[53289]: mds.? [v2:172.21.15.8:6834/511465636,v1:172.21.15.8:6835/511465636] up:active 2024-04-14T13:48:16.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:15 smithi012 ceph-mon[53289]: fsmap foofs:1 {0=foofs.smithi008.wqbvfi=up:active} 2024-04-14T13:48:16.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:15 smithi008 ceph-mon[67070]: pgmap v319: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 MiB/s rd, 383 B/s wr, 8 op/s 2024-04-14T13:48:16.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:15 smithi008 ceph-mon[67070]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-14T13:48:16.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:15 smithi008 ceph-mon[67070]: mds.? [v2:172.21.15.8:6834/511465636,v1:172.21.15.8:6835/511465636] up:active 2024-04-14T13:48:16.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:15 smithi008 ceph-mon[67070]: fsmap foofs:1 {0=foofs.smithi008.wqbvfi=up:active} 2024-04-14T13:48:18.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:17 smithi012 ceph-mon[53289]: pgmap v320: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 409 B/s wr, 8 op/s 2024-04-14T13:48:18.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:17 smithi008 ceph-mon[67070]: pgmap v320: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 409 B/s wr, 8 op/s 2024-04-14T13:48:19.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:19 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:19.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:19 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:48:19.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:19 smithi012 ceph-mon[53289]: pgmap v321: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 511 B/s wr, 10 op/s 2024-04-14T13:48:19.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:19 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:19.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:19 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:48:19.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:19 smithi008 ceph-mon[67070]: pgmap v321: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 511 B/s wr, 10 op/s 2024-04-14T13:48:21.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:20 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:21.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:20 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:21.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:20 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:48:21.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:20 smithi012 ceph-mon[53289]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-14T13:48:21.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:20 smithi012 ceph-mon[53289]: mds.? [v2:172.21.15.12:6832/3362773133,v1:172.21.15.12:6833/3362773133] up:boot 2024-04-14T13:48:21.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:20 smithi012 ceph-mon[53289]: fsmap foofs:1 {0=foofs.smithi008.wqbvfi=up:active} 1 up:standby 2024-04-14T13:48:21.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:20 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi012.vjovzp"}]: dispatch 2024-04-14T13:48:21.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:20 smithi012 ceph-mon[53289]: pgmap v322: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 307 B/s wr, 11 op/s 2024-04-14T13:48:21.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:20 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:21.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:20 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:21.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:20 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:48:21.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:20 smithi008 ceph-mon[67070]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-14T13:48:21.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:20 smithi008 ceph-mon[67070]: mds.? [v2:172.21.15.12:6832/3362773133,v1:172.21.15.12:6833/3362773133] up:boot 2024-04-14T13:48:21.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:20 smithi008 ceph-mon[67070]: fsmap foofs:1 {0=foofs.smithi008.wqbvfi=up:active} 1 up:standby 2024-04-14T13:48:21.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:20 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi012.vjovzp"}]: dispatch 2024-04-14T13:48:21.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:20 smithi008 ceph-mon[67070]: pgmap v322: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 307 B/s wr, 11 op/s 2024-04-14T13:48:24.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:23 smithi012 ceph-mon[53289]: pgmap v323: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 258 B/s wr, 9 op/s 2024-04-14T13:48:24.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:23 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:24.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:23 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:24.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:23 smithi008 ceph-mon[67070]: pgmap v323: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 258 B/s wr, 9 op/s 2024-04-14T13:48:24.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:23 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:24.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:23 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:25.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:25 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:25.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:25 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:25.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:25 smithi012 ceph-mon[53289]: pgmap v324: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 170 B/s wr, 6 op/s 2024-04-14T13:48:25.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:25 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:25.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:25 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:25.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:25 smithi008 ceph-mon[67070]: pgmap v324: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 170 B/s wr, 6 op/s 2024-04-14T13:48:28.012 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:27 smithi012 ceph-mon[53289]: pgmap v325: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 7.3 KiB/s wr, 6 op/s 2024-04-14T13:48:28.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:27 smithi008 ceph-mon[67070]: pgmap v325: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 7.3 KiB/s wr, 6 op/s 2024-04-14T13:48:29.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: Detected new or changed devices on smithi012 2024-04-14T13:48:29.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:29.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:29.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:48:29.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:48:29.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: pgmap v326: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7.8 KiB/s wr, 6 op/s 2024-04-14T13:48:29.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: pgmap v327: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 9.3 KiB/s wr, 5 op/s 2024-04-14T13:48:29.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:29.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:29.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:48:29.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: Upgrade: Setting container_image for all mds 2024-04-14T13:48:29.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:29.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi008.wqbvfi"}]: dispatch 2024-04-14T13:48:29.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi008.wqbvfi"}]': finished 2024-04-14T13:48:29.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi012.vjovzp"}]: dispatch 2024-04-14T13:48:29.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi012.vjovzp"}]': finished 2024-04-14T13:48:29.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: Upgrade: Setting container_image for all rgw 2024-04-14T13:48:29.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:29.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T13:48:29.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:29.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T13:48:29.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:29.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: Upgrade: Setting container_image for all iscsi 2024-04-14T13:48:29.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:29 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:29.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: Detected new or changed devices on smithi012 2024-04-14T13:48:29.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:29.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:29.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:48:29.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:48:29.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: pgmap v326: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7.8 KiB/s wr, 6 op/s 2024-04-14T13:48:29.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: pgmap v327: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 9.3 KiB/s wr, 5 op/s 2024-04-14T13:48:29.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:29.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:29.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:48:29.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: Upgrade: Setting container_image for all mds 2024-04-14T13:48:29.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:29.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi008.wqbvfi"}]: dispatch 2024-04-14T13:48:29.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi008.wqbvfi"}]': finished 2024-04-14T13:48:29.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi012.vjovzp"}]: dispatch 2024-04-14T13:48:29.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi012.vjovzp"}]': finished 2024-04-14T13:48:29.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: Upgrade: Setting container_image for all rgw 2024-04-14T13:48:29.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:29.552 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T13:48:29.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:29.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T13:48:29.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:29.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:29.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: Upgrade: Setting container_image for all iscsi 2024-04-14T13:48:29.553 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:29 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:31.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:31 smithi012 ceph-mon[53289]: Upgrade: Updating nfs.foo.0.0.smithi008.vefdqq (1/2) 2024-04-14T13:48:31.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:31.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:31 smithi012 ceph-mon[53289]: Creating key for client.nfs.foo.0.0.smithi008.vefdqq 2024-04-14T13:48:31.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.vefdqq", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-14T13:48:31.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:31 smithi012 ceph-mon[53289]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-14T13:48:31.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-14T13:48:31.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-14T13:48:31.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:48:31.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:31 smithi012 ceph-mon[53289]: pgmap v328: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 222 B/s rd, 9.3 KiB/s wr, 4 op/s 2024-04-14T13:48:31.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-14T13:48:31.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-14T13:48:31.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:31 smithi012 ceph-mon[53289]: Rados config object exists: conf-nfs.foo 2024-04-14T13:48:31.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:31 smithi012 ceph-mon[53289]: Creating key for client.nfs.foo.0.0.smithi008.vefdqq-rgw 2024-04-14T13:48:31.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.vefdqq-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T13:48:31.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:31 smithi012 ceph-mon[53289]: Bind address in nfs.foo.0.0.smithi008.vefdqq's ganesha conf is defaulting to empty 2024-04-14T13:48:31.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:48:31.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:31 smithi012 ceph-mon[53289]: Deploying daemon nfs.foo.0.0.smithi008.vefdqq on smithi008 2024-04-14T13:48:31.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:31 smithi008 ceph-mon[67070]: Upgrade: Updating nfs.foo.0.0.smithi008.vefdqq (1/2) 2024-04-14T13:48:31.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:31.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:31 smithi008 ceph-mon[67070]: Creating key for client.nfs.foo.0.0.smithi008.vefdqq 2024-04-14T13:48:31.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.vefdqq", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-14T13:48:31.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:31 smithi008 ceph-mon[67070]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-14T13:48:31.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-14T13:48:31.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-14T13:48:31.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:48:31.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:31 smithi008 ceph-mon[67070]: pgmap v328: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 222 B/s rd, 9.3 KiB/s wr, 4 op/s 2024-04-14T13:48:31.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-14T13:48:31.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-14T13:48:31.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:31 smithi008 ceph-mon[67070]: Rados config object exists: conf-nfs.foo 2024-04-14T13:48:31.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:31 smithi008 ceph-mon[67070]: Creating key for client.nfs.foo.0.0.smithi008.vefdqq-rgw 2024-04-14T13:48:31.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.vefdqq-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T13:48:31.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:31 smithi008 ceph-mon[67070]: Bind address in nfs.foo.0.0.smithi008.vefdqq's ganesha conf is defaulting to empty 2024-04-14T13:48:31.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:48:31.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:31 smithi008 ceph-mon[67070]: Deploying daemon nfs.foo.0.0.smithi008.vefdqq on smithi008 2024-04-14T13:48:33.276 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:33 smithi008 ceph-mon[67070]: pgmap v329: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 222 B/s rd, 9.3 KiB/s wr, 4 op/s 2024-04-14T13:48:33.277 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:33 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:33.277 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:33 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:48:33.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:33 smithi012 ceph-mon[53289]: pgmap v329: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 222 B/s rd, 9.3 KiB/s wr, 4 op/s 2024-04-14T13:48:33.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:33 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:33.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:33 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:48:34.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:34 smithi012 ceph-mon[53289]: from='mds.? [v2:172.21.15.8:6834/511465636,v1:172.21.15.8:6835/511465636]' entity='mds.foofs.smithi008.wqbvfi' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.8:0/2332098306"}]: dispatch 2024-04-14T13:48:34.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:34 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:34.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:34 smithi008 ceph-mon[67070]: from='mds.? [v2:172.21.15.8:6834/511465636,v1:172.21.15.8:6835/511465636]' entity='mds.foofs.smithi008.wqbvfi' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.8:0/2332098306"}]: dispatch 2024-04-14T13:48:34.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:34 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:35.451 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:35 smithi012 ceph-mon[53289]: Evicting (and blocklisting) client session 14560 (172.21.15.8:0/2332098306) 2024-04-14T13:48:35.451 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:35 smithi012 ceph-mon[53289]: pgmap v330: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 9.8 KiB/s wr, 6 op/s 2024-04-14T13:48:35.451 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:35 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:35.451 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:35 smithi012 ceph-mon[53289]: from='mds.? [v2:172.21.15.8:6834/511465636,v1:172.21.15.8:6835/511465636]' entity='mds.foofs.smithi008.wqbvfi' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.8:0/2332098306"}]': finished 2024-04-14T13:48:35.451 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:35 smithi012 ceph-mon[53289]: osdmap e153: 8 total, 8 up, 8 in 2024-04-14T13:48:35.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:35 smithi008 ceph-mon[67070]: Evicting (and blocklisting) client session 14560 (172.21.15.8:0/2332098306) 2024-04-14T13:48:35.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:35 smithi008 ceph-mon[67070]: pgmap v330: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 9.8 KiB/s wr, 6 op/s 2024-04-14T13:48:35.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:35 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:35.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:35 smithi008 ceph-mon[67070]: from='mds.? [v2:172.21.15.8:6834/511465636,v1:172.21.15.8:6835/511465636]' entity='mds.foofs.smithi008.wqbvfi' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.8:0/2332098306"}]': finished 2024-04-14T13:48:35.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:35 smithi008 ceph-mon[67070]: osdmap e153: 8 total, 8 up, 8 in 2024-04-14T13:48:37.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:36 smithi008 ceph-mon[67070]: Upgrade: Updating nfs.foo.1.0.smithi012.meswmn (2/2) 2024-04-14T13:48:37.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:36 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:37.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:36 smithi008 ceph-mon[67070]: Creating key for client.nfs.foo.1.0.smithi012.meswmn 2024-04-14T13:48:37.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:36 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi012.meswmn", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-14T13:48:37.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:36 smithi008 ceph-mon[67070]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-14T13:48:37.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:36 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-14T13:48:37.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:36 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-14T13:48:37.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:36 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:48:37.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:36 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-14T13:48:37.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:36 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-14T13:48:37.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:36 smithi008 ceph-mon[67070]: Rados config object exists: conf-nfs.foo 2024-04-14T13:48:37.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:36 smithi008 ceph-mon[67070]: Creating key for client.nfs.foo.1.0.smithi012.meswmn-rgw 2024-04-14T13:48:37.107 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:36 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi012.meswmn-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T13:48:37.107 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:36 smithi008 ceph-mon[67070]: Bind address in nfs.foo.1.0.smithi012.meswmn's ganesha conf is defaulting to empty 2024-04-14T13:48:37.107 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:36 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:48:37.107 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:36 smithi008 ceph-mon[67070]: Deploying daemon nfs.foo.1.0.smithi012.meswmn on smithi012 2024-04-14T13:48:37.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:36 smithi012 ceph-mon[53289]: Upgrade: Updating nfs.foo.1.0.smithi012.meswmn (2/2) 2024-04-14T13:48:37.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:36 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:37.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:36 smithi012 ceph-mon[53289]: Creating key for client.nfs.foo.1.0.smithi012.meswmn 2024-04-14T13:48:37.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:36 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi012.meswmn", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-14T13:48:37.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:36 smithi012 ceph-mon[53289]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-14T13:48:37.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:36 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-14T13:48:37.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:36 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-14T13:48:37.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:36 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:48:37.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:36 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-14T13:48:37.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:36 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-14T13:48:37.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:36 smithi012 ceph-mon[53289]: Rados config object exists: conf-nfs.foo 2024-04-14T13:48:37.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:36 smithi012 ceph-mon[53289]: Creating key for client.nfs.foo.1.0.smithi012.meswmn-rgw 2024-04-14T13:48:37.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:36 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi012.meswmn-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T13:48:37.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:36 smithi012 ceph-mon[53289]: Bind address in nfs.foo.1.0.smithi012.meswmn's ganesha conf is defaulting to empty 2024-04-14T13:48:37.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:36 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:48:37.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:36 smithi012 ceph-mon[53289]: Deploying daemon nfs.foo.1.0.smithi012.meswmn on smithi012 2024-04-14T13:48:38.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:37 smithi012 ceph-mon[53289]: pgmap v332: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-14T13:48:38.204 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:37 smithi008 ceph-mon[67070]: pgmap v332: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-14T13:48:39.196 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:48:39.879 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:48:39.879 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (10m) 39s ago 19m 24.9M - 0.23.0 ba2b418f427c cfddb9a7f858 2024-04-14T13:48:39.879 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (6m) 39s ago 19m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 1e69eabdcc36 2024-04-14T13:48:39.879 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (6m) 16s ago 18m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 60c7a2992eb0 2024-04-14T13:48:39.879 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 39s ago 19m - - 2024-04-14T13:48:39.879 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (44s) 39s ago 13m 11.2M - 19.0.0-2778-ga971fd5f b96c509eef84 50aacbc1bc98 2024-04-14T13:48:39.879 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (20s) 16s ago 13m 13.8M - 19.0.0-2778-ga971fd5f b96c509eef84 cdb17d8fe5ff 2024-04-14T13:48:39.879 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (9m) 39s ago 20m 512M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:48:39.879 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (8m) 16s ago 18m 454M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:48:39.880 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (8m) 39s ago 21m 59.5M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 673dd6f4c8d8 2024-04-14T13:48:39.880 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (7m) 16s ago 18m 47.0M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 d498dec29c4f 2024-04-14T13:48:39.880 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 starting - - - - 2024-04-14T13:48:39.880 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 starting - - - - 2024-04-14T13:48:39.880 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (10m) 39s ago 19m 17.4M - 1.3.1 1dbe0e931976 646dfca70be4 2024-04-14T13:48:39.880 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (9m) 16s ago 18m 19.2M - 1.3.1 1dbe0e931976 0a60dcd4044c 2024-04-14T13:48:39.880 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (5m) 39s ago 18m 392M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 29997ccc66ac 2024-04-14T13:48:39.880 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (5m) 39s ago 17m 361M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 97df88f10bd6 2024-04-14T13:48:39.880 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (4m) 39s ago 17m 324M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 9325b527c16c 2024-04-14T13:48:39.880 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (3m) 39s ago 16m 427M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 6868b5076d61 2024-04-14T13:48:39.880 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (3m) 16s ago 16m 365M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 178f9462f5ba 2024-04-14T13:48:39.880 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (2m) 16s ago 15m 316M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 25e416621d40 2024-04-14T13:48:39.880 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (104s) 16s ago 15m 292M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 97f0b691cf87 2024-04-14T13:48:39.880 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (66s) 16s ago 15m 288M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 7e4e731dec31 2024-04-14T13:48:39.880 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (8m) 39s ago 18m 70.5M - 2.33.4 514e6a882f6e 12f97f75d620 2024-04-14T13:48:40.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:39 smithi012 ceph-mon[53289]: pgmap v333: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-14T13:48:40.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:39 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:40.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:39 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:40.325 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:39 smithi008 ceph-mon[67070]: pgmap v333: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-14T13:48:40.325 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:39 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:40.325 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:39 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:40.325 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:48:40.325 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:48:40.325 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:48:40.326 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:48:40.326 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:48:40.326 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:48:40.326 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:48:40.326 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:48:40.326 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 8 2024-04-14T13:48:40.326 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:48:40.326 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:48:40.326 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:48:40.326 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:48:40.326 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:48:40.326 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 14 2024-04-14T13:48:40.326 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:48:40.326 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:48:40.696 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:48:40.697 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:48:40.697 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:48:40.697 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:48:40.697 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-14T13:48:40.697 INFO:teuthology.orchestra.run.smithi008.stdout: "osd", 2024-04-14T13:48:40.697 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-04-14T13:48:40.697 INFO:teuthology.orchestra.run.smithi008.stdout: "mds", 2024-04-14T13:48:40.697 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-04-14T13:48:40.697 INFO:teuthology.orchestra.run.smithi008.stdout: "crash" 2024-04-14T13:48:40.697 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:48:40.697 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "16/23 daemons upgraded", 2024-04-14T13:48:40.697 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading nfs daemons", 2024-04-14T13:48:40.697 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:48:40.697 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:48:40.996 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:40 smithi012 ceph-mon[53289]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:48:40.996 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:40 smithi012 ceph-mon[53289]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:48:40.996 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:40 smithi012 ceph-mon[53289]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:48:40.996 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:40 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:48:40.996 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:40 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/2232880866' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:41.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:40 smithi008 ceph-mon[67070]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:48:41.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:40 smithi008 ceph-mon[67070]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:48:41.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:40 smithi008 ceph-mon[67070]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:48:41.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:40 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:48:41.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:40 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/2232880866' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:41.119 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T13:48:41.119 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:48:41.119 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-14T13:48:42.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:41 smithi012 ceph-mon[53289]: pgmap v334: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-04-14T13:48:42.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:41 smithi012 ceph-mon[53289]: from='client.34552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:48:42.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:41 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/2717780308' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:48:42.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:41 smithi008 ceph-mon[67070]: pgmap v334: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-04-14T13:48:42.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:41 smithi008 ceph-mon[67070]: from='client.34552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:48:42.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:41 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/2717780308' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:48:44.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:43 smithi008 ceph-mon[67070]: pgmap v335: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-04-14T13:48:44.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:43 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:44.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:43 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:44.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:43 smithi008 ceph-mon[67070]: Evicting (and blocklisting) client session 24373 (172.21.15.12:0/1442085824) 2024-04-14T13:48:44.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:43 smithi008 ceph-mon[67070]: from='mds.? [v2:172.21.15.8:6834/511465636,v1:172.21.15.8:6835/511465636]' entity='mds.foofs.smithi008.wqbvfi' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.12:0/1442085824"}]: dispatch 2024-04-14T13:48:44.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:43 smithi012 ceph-mon[53289]: pgmap v335: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-04-14T13:48:44.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:43 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:44.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:43 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:44.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:43 smithi012 ceph-mon[53289]: Evicting (and blocklisting) client session 24373 (172.21.15.12:0/1442085824) 2024-04-14T13:48:44.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:43 smithi012 ceph-mon[53289]: from='mds.? [v2:172.21.15.8:6834/511465636,v1:172.21.15.8:6835/511465636]' entity='mds.foofs.smithi008.wqbvfi' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.12:0/1442085824"}]: dispatch 2024-04-14T13:48:44.976 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:44 smithi012 ceph-mon[53289]: from='mds.? [v2:172.21.15.8:6834/511465636,v1:172.21.15.8:6835/511465636]' entity='mds.foofs.smithi008.wqbvfi' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.12:0/1442085824"}]': finished 2024-04-14T13:48:44.976 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:44 smithi012 ceph-mon[53289]: osdmap e154: 8 total, 8 up, 8 in 2024-04-14T13:48:44.976 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:44.976 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:44.976 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:44.976 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:45.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:44 smithi008 ceph-mon[67070]: from='mds.? [v2:172.21.15.8:6834/511465636,v1:172.21.15.8:6835/511465636]' entity='mds.foofs.smithi008.wqbvfi' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.12:0/1442085824"}]': finished 2024-04-14T13:48:45.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:44 smithi008 ceph-mon[67070]: osdmap e154: 8 total, 8 up, 8 in 2024-04-14T13:48:45.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:45.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:45.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:45.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:46.036 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:45 smithi012 ceph-mon[53289]: pgmap v337: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-14T13:48:46.036 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:45 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:46.036 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:45 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:46.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:45 smithi008 ceph-mon[67070]: pgmap v337: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-14T13:48:46.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:45 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:46.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:45 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:48.081 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:47 smithi008 ceph-mon[67070]: pgmap v338: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-14T13:48:48.081 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:47 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:48.081 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:47 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:48.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:47 smithi012 ceph-mon[53289]: pgmap v338: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-14T13:48:48.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:47 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:48.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:47 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:49.294 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:48:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:48:50.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: pgmap v339: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-14T13:48:50.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:50.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:50.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:48:50.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:48:50.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:50.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:50.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:48:50.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:50.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi008.vefdqq"}]: dispatch 2024-04-14T13:48:50.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi008.vefdqq"}]': finished 2024-04-14T13:48:50.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi012.meswmn"}]: dispatch 2024-04-14T13:48:50.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi012.meswmn"}]': finished 2024-04-14T13:48:50.303 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.303 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:49 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:50.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: pgmap v339: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-14T13:48:50.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:50.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:50.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:48:50.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:48:50.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:50.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:50.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:48:50.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:50.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi008.vefdqq"}]: dispatch 2024-04-14T13:48:50.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi008.vefdqq"}]': finished 2024-04-14T13:48:50.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi012.meswmn"}]: dispatch 2024-04-14T13:48:50.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi012.meswmn"}]': finished 2024-04-14T13:48:50.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:48:50.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:49 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:51.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:50 smithi008 ceph-mon[67070]: pgmap v340: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-14T13:48:51.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:50 smithi008 ceph-mon[67070]: pgmap v341: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-14T13:48:51.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:50 smithi008 ceph-mon[67070]: Upgrade: Setting container_image for all nfs 2024-04-14T13:48:51.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:50 smithi008 ceph-mon[67070]: Upgrade: Setting container_image for all nvmeof 2024-04-14T13:48:51.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:50 smithi012 ceph-mon[53289]: pgmap v340: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-14T13:48:51.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:50 smithi012 ceph-mon[53289]: pgmap v341: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-14T13:48:51.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:50 smithi012 ceph-mon[53289]: Upgrade: Setting container_image for all nfs 2024-04-14T13:48:51.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:50 smithi012 ceph-mon[53289]: Upgrade: Setting container_image for all nvmeof 2024-04-14T13:48:53.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:52 smithi012 ceph-mon[53289]: Upgrade: Updating node-exporter.smithi008 (1/2) 2024-04-14T13:48:53.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:52 smithi012 ceph-mon[53289]: Deploying daemon node-exporter.smithi008 on smithi008 2024-04-14T13:48:53.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:52 smithi012 ceph-mon[53289]: pgmap v342: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-14T13:48:53.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:52 smithi008 ceph-mon[67070]: Upgrade: Updating node-exporter.smithi008 (1/2) 2024-04-14T13:48:53.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:52 smithi008 ceph-mon[67070]: Deploying daemon node-exporter.smithi008 on smithi008 2024-04-14T13:48:53.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:52 smithi008 ceph-mon[67070]: pgmap v342: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-14T13:48:55.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:54 smithi012 ceph-mon[53289]: pgmap v343: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 426 B/s wr, 0 op/s 2024-04-14T13:48:55.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:54 smithi008 ceph-mon[67070]: pgmap v343: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 426 B/s wr, 0 op/s 2024-04-14T13:48:57.057 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:56 smithi008 ceph-mon[67070]: pgmap v344: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 106 B/s rd, 0 B/s wr, 0 op/s 2024-04-14T13:48:57.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:56 smithi012 ceph-mon[53289]: pgmap v344: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 106 B/s rd, 0 B/s wr, 0 op/s 2024-04-14T13:48:58.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:58 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:58.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:58 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:58.551 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:48:58 smithi012 ceph-mon[53289]: pgmap v345: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 106 B/s rd, 0 op/s 2024-04-14T13:48:58.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:58 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:58.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:58 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:48:58.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:48:58 smithi008 ceph-mon[67070]: pgmap v345: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 106 B/s rd, 0 op/s 2024-04-14T13:49:01.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:00 smithi012 ceph-mon[53289]: Upgrade: Updating node-exporter.smithi012 (2/2) 2024-04-14T13:49:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:00 smithi012 ceph-mon[53289]: Deploying daemon node-exporter.smithi012 on smithi012 2024-04-14T13:49:01.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:00 smithi012 ceph-mon[53289]: pgmap v346: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-14T13:49:01.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:00 smithi008 ceph-mon[67070]: Upgrade: Updating node-exporter.smithi012 (2/2) 2024-04-14T13:49:01.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:00 smithi008 ceph-mon[67070]: Deploying daemon node-exporter.smithi012 on smithi012 2024-04-14T13:49:01.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:00 smithi008 ceph-mon[67070]: pgmap v346: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-14T13:49:03.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:02 smithi012 ceph-mon[53289]: pgmap v347: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:49:03.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:02 smithi008 ceph-mon[67070]: pgmap v347: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:49:03.919 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:49:04.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:49:05.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:04 smithi012 ceph-mon[53289]: pgmap v348: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:49:05.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:04 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:05.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:04 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:05.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:04 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:49:05.054 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:04 smithi008 ceph-mon[67070]: pgmap v348: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:49:05.054 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:04 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:05.054 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:04 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:05.054 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:04 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:49:07.007 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:06 smithi012 ceph-mon[53289]: pgmap v349: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-14T13:49:07.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:06 smithi008 ceph-mon[67070]: pgmap v349: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-14T13:49:08.537 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:08 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:08.537 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:08 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:08.537 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:08 smithi008 ceph-mon[67070]: pgmap v350: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:49:08.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:08 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:08.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:08 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:08.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:08 smithi012 ceph-mon[53289]: pgmap v350: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:49:09.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:09 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:09.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:09 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:09.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:09 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:09.550 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:09 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:09.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:09 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:09.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:09 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:09.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:09 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:09.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:09 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:11.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:10 smithi008 ceph-mon[67070]: pgmap v351: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-14T13:49:11.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:10 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:11.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:10 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:11.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:10 smithi012 ceph-mon[53289]: pgmap v351: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-14T13:49:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:10 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:11.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:10 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:11.480 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:49:12.189 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:49:12.189 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (11m) 3s ago 20m 24.4M - 0.23.0 ba2b418f427c cfddb9a7f858 2024-04-14T13:49:12.189 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (7m) 3s ago 20m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 1e69eabdcc36 2024-04-14T13:49:12.189 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (6m) 4s ago 19m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 60c7a2992eb0 2024-04-14T13:49:12.189 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 3s ago 19m - - 2024-04-14T13:49:12.189 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (77s) 3s ago 14m 34.8M - 19.0.0-2778-ga971fd5f b96c509eef84 50aacbc1bc98 2024-04-14T13:49:12.189 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (52s) 4s ago 14m 14.0M - 19.0.0-2778-ga971fd5f b96c509eef84 cdb17d8fe5ff 2024-04-14T13:49:12.190 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (9m) 3s ago 21m 514M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:49:12.190 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (8m) 4s ago 19m 454M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:49:12.190 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (8m) 3s ago 21m 62.4M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 673dd6f4c8d8 2024-04-14T13:49:12.190 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (8m) 4s ago 19m 48.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 d498dec29c4f 2024-04-14T13:49:12.190 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (38s) 3s ago 13m 72.3M - 5.7 b96c509eef84 0fc0feda12be 2024-04-14T13:49:12.190 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (32s) 4s ago 13m 68.2M - 5.7 b96c509eef84 56b616ccb6ed 2024-04-14T13:49:12.190 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (15s) 3s ago 19m 8514k - 1.5.0 0da6a335fe13 17213ca85724 2024-04-14T13:49:12.190 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (8s) 4s ago 19m 3363k - 1.5.0 0da6a335fe13 df7b43405919 2024-04-14T13:49:12.190 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (6m) 3s ago 18m 393M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 29997ccc66ac 2024-04-14T13:49:12.190 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (5m) 3s ago 18m 361M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 97df88f10bd6 2024-04-14T13:49:12.190 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (4m) 3s ago 17m 325M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 9325b527c16c 2024-04-14T13:49:12.190 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (4m) 3s ago 17m 429M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 6868b5076d61 2024-04-14T13:49:12.190 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (3m) 4s ago 16m 366M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 178f9462f5ba 2024-04-14T13:49:12.190 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (2m) 4s ago 16m 317M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 25e416621d40 2024-04-14T13:49:12.190 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (2m) 4s ago 16m 293M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 97f0b691cf87 2024-04-14T13:49:12.191 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (99s) 4s ago 15m 289M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 7e4e731dec31 2024-04-14T13:49:12.191 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (9m) 3s ago 18m 74.6M - 2.33.4 514e6a882f6e 12f97f75d620 2024-04-14T13:49:12.626 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:49:12.626 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:49:12.627 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:49:12.627 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:49:12.627 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:49:12.627 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:49:12.627 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:49:12.627 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:49:12.627 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 8 2024-04-14T13:49:12.627 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:49:12.627 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:49:12.627 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:49:12.627 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:49:12.627 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:49:12.627 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 14 2024-04-14T13:49:12.627 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:49:12.627 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:49:12.981 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:49:12.981 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:49:12.981 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:49:12.981 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:49:12.981 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-14T13:49:12.981 INFO:teuthology.orchestra.run.smithi008.stdout: "osd", 2024-04-14T13:49:12.981 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-04-14T13:49:12.981 INFO:teuthology.orchestra.run.smithi008.stdout: "nfs", 2024-04-14T13:49:12.981 INFO:teuthology.orchestra.run.smithi008.stdout: "mds", 2024-04-14T13:49:12.981 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-04-14T13:49:12.981 INFO:teuthology.orchestra.run.smithi008.stdout: "crash" 2024-04-14T13:49:12.981 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:49:12.981 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "18/23 daemons upgraded", 2024-04-14T13:49:12.981 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading node-exporter daemons", 2024-04-14T13:49:12.982 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:49:12.982 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:49:12.982 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:12 smithi008 ceph-mon[67070]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:49:12.982 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:12 smithi008 ceph-mon[67070]: pgmap v352: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:49:12.982 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:12 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:12.982 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:12 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:12.982 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:12 smithi008 ceph-mon[67070]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:49:12.982 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:12 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/3120940644' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:13.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:12 smithi012 ceph-mon[53289]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:49:13.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:12 smithi012 ceph-mon[53289]: pgmap v352: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:49:13.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:12 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:13.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:12 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:13.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:12 smithi012 ceph-mon[53289]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:49:13.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:12 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/3120940644' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:13.404 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T13:49:13.405 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:49:13.405 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-14T13:49:14.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:49:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='client.34580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:49:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/3307032932' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:49:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:49:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:49:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:49:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:13 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:49:14.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='client.34580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:49:14.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/3307032932' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:49:14.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:14.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:14.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:49:14.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:49:14.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:14.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:14.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:49:14.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:13 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:14 smithi008 ceph-mon[67070]: pgmap v353: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 172 B/s rd, 0 op/s 2024-04-14T13:49:14.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:14 smithi008 ceph-mon[67070]: pgmap v354: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 206 B/s rd, 0 op/s 2024-04-14T13:49:14.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:14 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:14 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:14.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:14 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:15.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:14 smithi012 ceph-mon[53289]: pgmap v353: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 172 B/s rd, 0 op/s 2024-04-14T13:49:15.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:14 smithi012 ceph-mon[53289]: pgmap v354: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 206 B/s rd, 0 op/s 2024-04-14T13:49:15.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:14 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:15.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:14 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:15.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:14 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:17.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:16 smithi012 ceph-mon[53289]: Upgrade: Updating prometheus.smithi008 2024-04-14T13:49:17.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:16 smithi012 ceph-mon[53289]: Deploying daemon prometheus.smithi008 on smithi008 2024-04-14T13:49:17.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:16 smithi012 ceph-mon[53289]: pgmap v355: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 206 B/s rd, 0 op/s 2024-04-14T13:49:17.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:16 smithi008 ceph-mon[67070]: Upgrade: Updating prometheus.smithi008 2024-04-14T13:49:17.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:16 smithi008 ceph-mon[67070]: Deploying daemon prometheus.smithi008 on smithi008 2024-04-14T13:49:17.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:16 smithi008 ceph-mon[67070]: pgmap v355: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 206 B/s rd, 0 op/s 2024-04-14T13:49:18.942 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:18 smithi008 ceph-mon[67070]: pgmap v356: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 206 B/s rd, 0 op/s 2024-04-14T13:49:18.942 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:18 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:49:19.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:18 smithi012 ceph-mon[53289]: pgmap v356: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 206 B/s rd, 0 op/s 2024-04-14T13:49:19.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:18 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:49:21.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:20 smithi012 ceph-mon[53289]: pgmap v357: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 206 B/s rd, 0 op/s 2024-04-14T13:49:21.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:20 smithi008 ceph-mon[67070]: pgmap v357: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 206 B/s rd, 0 op/s 2024-04-14T13:49:22.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:22 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:22.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:22 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:22.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:22 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:49:22.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:22 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:22.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:22 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:22.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:22 smithi012 ceph-mon[53289]: pgmap v358: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 206 B/s rd, 0 op/s 2024-04-14T13:49:22.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:22 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:22.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:22 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:22.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:22 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:49:22.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:22 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:22.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:22 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:22.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:22 smithi008 ceph-mon[67070]: pgmap v358: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 206 B/s rd, 0 op/s 2024-04-14T13:49:24.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:24 smithi008 ceph-mon[67070]: pgmap v359: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-14T13:49:25.004 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:24 smithi012 ceph-mon[53289]: pgmap v359: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-14T13:49:26.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:26 smithi012 ceph-mon[53289]: pgmap v360: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:49:26.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:26 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:26.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:26 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:27.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:26 smithi008 ceph-mon[67070]: pgmap v360: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:49:27.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:26 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:27.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:26 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:28.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:28 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:28.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:28 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:28.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:28 smithi008 ceph-mon[67070]: pgmap v361: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-14T13:49:28.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:28 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:28.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:28 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:28.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:28 smithi012 ceph-mon[53289]: pgmap v361: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-14T13:49:30.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:30 smithi008 ceph-mon[67070]: pgmap v362: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-14T13:49:31.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:30 smithi012 ceph-mon[53289]: pgmap v362: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-14T13:49:32.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:32.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:32.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:49:32.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:49:32.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: pgmap v363: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 183 B/s rd, 0 op/s 2024-04-14T13:49:32.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: pgmap v364: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 223 B/s rd, 0 op/s 2024-04-14T13:49:32.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:32.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:32.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:49:32.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:49:32.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.036 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:31 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:32.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:32.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:49:32.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:49:32.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: pgmap v363: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 183 B/s rd, 0 op/s 2024-04-14T13:49:32.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: pgmap v364: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 223 B/s rd, 0 op/s 2024-04-14T13:49:32.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:32.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:32.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:49:32.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:49:32.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:32.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:31 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:33.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:32 smithi012 ceph-mon[53289]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:49:33.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:32 smithi008 ceph-mon[67070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:49:34.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:33 smithi012 ceph-mon[53289]: Upgrade: Updating alertmanager.smithi008 2024-04-14T13:49:34.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:33 smithi012 ceph-mon[53289]: Deploying daemon alertmanager.smithi008 on smithi008 2024-04-14T13:49:34.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:33 smithi012 ceph-mon[53289]: pgmap v365: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 223 B/s rd, 0 op/s 2024-04-14T13:49:34.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:33 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:49:34.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:33 smithi008 ceph-mon[67070]: Upgrade: Updating alertmanager.smithi008 2024-04-14T13:49:34.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:33 smithi008 ceph-mon[67070]: Deploying daemon alertmanager.smithi008 on smithi008 2024-04-14T13:49:34.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:33 smithi008 ceph-mon[67070]: pgmap v365: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 223 B/s rd, 0 op/s 2024-04-14T13:49:34.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:33 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:49:36.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:35 smithi012 ceph-mon[53289]: pgmap v366: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 223 B/s rd, 0 op/s 2024-04-14T13:49:36.055 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:35 smithi008 ceph-mon[67070]: pgmap v366: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 223 B/s rd, 0 op/s 2024-04-14T13:49:37.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:37 smithi008 ceph-mon[67070]: pgmap v367: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 223 B/s rd, 0 op/s 2024-04-14T13:49:38.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:37 smithi012 ceph-mon[53289]: pgmap v367: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 223 B/s rd, 0 op/s 2024-04-14T13:49:39.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:39 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:39.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:39 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:39.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:39 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:49:39.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:39 smithi008 ceph-mon[67070]: pgmap v368: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 223 B/s rd, 0 op/s 2024-04-14T13:49:39.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:39 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:39.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:39 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:39 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:39 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:39 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:49:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:39 smithi012 ceph-mon[53289]: pgmap v368: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 223 B/s rd, 0 op/s 2024-04-14T13:49:39.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:39 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:39.801 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:39 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:42.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:41 smithi008 ceph-mon[67070]: pgmap v369: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-14T13:49:42.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:41 smithi012 ceph-mon[53289]: pgmap v369: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-14T13:49:43.772 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:49:44.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:43 smithi008 ceph-mon[67070]: pgmap v370: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:49:44.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:43 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:44.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:43 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:44.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:43 smithi012 ceph-mon[53289]: pgmap v370: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:49:44.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:43 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:44.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:43 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:44.492 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:49:44.492 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (6s) 1s ago 20m 16.6M - 0.25.0 c8568f914cd2 77146b15f08c 2024-04-14T13:49:44.492 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (7m) 1s ago 20m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 1e69eabdcc36 2024-04-14T13:49:44.492 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (7m) 37s ago 20m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 60c7a2992eb0 2024-04-14T13:49:44.492 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 1s ago 20m - - 2024-04-14T13:49:44.492 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (109s) 1s ago 14m 35.1M - 19.0.0-2778-ga971fd5f b96c509eef84 50aacbc1bc98 2024-04-14T13:49:44.492 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (84s) 37s ago 14m 14.0M - 19.0.0-2778-ga971fd5f b96c509eef84 cdb17d8fe5ff 2024-04-14T13:49:44.492 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (10m) 1s ago 21m 516M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:49:44.493 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (9m) 37s ago 19m 454M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:49:44.493 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (9m) 1s ago 22m 62.6M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 673dd6f4c8d8 2024-04-14T13:49:44.493 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (8m) 37s ago 19m 48.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 d498dec29c4f 2024-04-14T13:49:44.493 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (70s) 1s ago 14m 72.6M - 5.7 b96c509eef84 0fc0feda12be 2024-04-14T13:49:44.493 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (65s) 37s ago 14m 68.2M - 5.7 b96c509eef84 56b616ccb6ed 2024-04-14T13:49:44.493 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (47s) 1s ago 20m 8108k - 1.5.0 0da6a335fe13 17213ca85724 2024-04-14T13:49:44.493 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (40s) 37s ago 19m 3363k - 1.5.0 0da6a335fe13 df7b43405919 2024-04-14T13:49:44.493 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (6m) 1s ago 19m 393M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 29997ccc66ac 2024-04-14T13:49:44.493 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (6m) 1s ago 18m 362M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 97df88f10bd6 2024-04-14T13:49:44.493 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (5m) 1s ago 18m 326M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 9325b527c16c 2024-04-14T13:49:44.493 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (4m) 1s ago 17m 429M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 6868b5076d61 2024-04-14T13:49:44.493 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (4m) 37s ago 17m 366M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 178f9462f5ba 2024-04-14T13:49:44.493 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (3m) 37s ago 17m 317M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 25e416621d40 2024-04-14T13:49:44.493 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (2m) 37s ago 16m 293M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 97f0b691cf87 2024-04-14T13:49:44.493 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (2m) 37s ago 16m 289M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 7e4e731dec31 2024-04-14T13:49:44.494 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (23s) 1s ago 19m 53.0M - 2.43.0 a07b618ecd1d 492f8cdc3836 2024-04-14T13:49:44.921 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:49:44.921 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:49:44.921 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:49:44.921 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:49:44.921 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:49:44.921 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:49:44.921 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:49:44.922 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:49:44.922 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 8 2024-04-14T13:49:44.922 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:49:44.922 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:49:44.922 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:49:44.922 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:49:44.922 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:49:44.922 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 14 2024-04-14T13:49:44.922 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:49:44.922 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:49:45.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:44 smithi008 ceph-mon[67070]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:49:45.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:45.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:44 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:45.284 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:49:45.284 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", 2024-04-14T13:49:45.284 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-14T13:49:45.284 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T13:49:45.284 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-14T13:49:45.284 INFO:teuthology.orchestra.run.smithi008.stdout: "osd", 2024-04-14T13:49:45.284 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-04-14T13:49:45.284 INFO:teuthology.orchestra.run.smithi008.stdout: "nfs", 2024-04-14T13:49:45.284 INFO:teuthology.orchestra.run.smithi008.stdout: "mds", 2024-04-14T13:49:45.284 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-04-14T13:49:45.284 INFO:teuthology.orchestra.run.smithi008.stdout: "crash" 2024-04-14T13:49:45.285 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:49:45.285 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "18/23 daemons upgraded", 2024-04-14T13:49:45.285 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading alertmanager daemons", 2024-04-14T13:49:45.285 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:49:45.285 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:49:45.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:44 smithi012 ceph-mon[53289]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:49:45.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:45.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:44 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:45.699 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T13:49:45.699 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T13:49:45.699 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-14T13:49:46.007 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:45 smithi008 ceph-mon[67070]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:49:46.007 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:45 smithi008 ceph-mon[67070]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:49:46.007 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:45 smithi008 ceph-mon[67070]: pgmap v371: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:49:46.007 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:45 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/189398072' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:46.007 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:45 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/1947511413' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:49:46.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:45 smithi012 ceph-mon[53289]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:49:46.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:45 smithi012 ceph-mon[53289]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:49:46.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:45 smithi012 ceph-mon[53289]: pgmap v371: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:49:46.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:45 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/189398072' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:46.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:45 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/1947511413' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:49:47.018 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:46 smithi008 ceph-mon[67070]: from='client.34604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:49:47.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:46 smithi012 ceph-mon[53289]: from='client.34604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:49:48.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:47 smithi008 ceph-mon[67070]: pgmap v372: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:49:48.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:47 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:48.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:47 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:48.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:47 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:49:48.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:47 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:49:48.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:47 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:48.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:47 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:48.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:47 smithi012 ceph-mon[53289]: pgmap v372: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:49:48.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:47 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:48.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:47 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:48.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:47 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:49:48.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:47 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:49:48.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:47 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:48.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:47 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:49.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: pgmap v373: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 186 B/s rd, 0 op/s 2024-04-14T13:49:49.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: pgmap v374: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 228 B/s rd, 0 op/s 2024-04-14T13:49:49.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T13:49:49.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T13:49:49.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:49:49.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:49:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: pgmap v373: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 186 B/s rd, 0 op/s 2024-04-14T13:49:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: pgmap v374: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 228 B/s rd, 0 op/s 2024-04-14T13:49:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T13:49:49.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T13:49:49.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:49:49.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:49:49.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:49:51.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:50 smithi008 ceph-mon[67070]: Upgrade: Updating grafana.smithi008 2024-04-14T13:49:51.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:50 smithi008 ceph-mon[67070]: Deploying daemon grafana.smithi008 on smithi008 2024-04-14T13:49:51.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:50 smithi008 ceph-mon[67070]: pgmap v375: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 228 B/s rd, 0 op/s 2024-04-14T13:49:51.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:50 smithi012 ceph-mon[53289]: Upgrade: Updating grafana.smithi008 2024-04-14T13:49:51.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:50 smithi012 ceph-mon[53289]: Deploying daemon grafana.smithi008 on smithi008 2024-04-14T13:49:51.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:50 smithi012 ceph-mon[53289]: pgmap v375: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 228 B/s rd, 0 op/s 2024-04-14T13:49:53.085 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:52 smithi008 ceph-mon[67070]: pgmap v376: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 228 B/s rd, 0 op/s 2024-04-14T13:49:53.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:52 smithi012 ceph-mon[53289]: pgmap v376: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 228 B/s rd, 0 op/s 2024-04-14T13:49:54.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:54 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:54.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:54 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:54.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:54 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:49:54.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:54 smithi008 ceph-mon[67070]: pgmap v377: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-14T13:49:54.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:54 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:54.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:54 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:54.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:54 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:49:54.800 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:54 smithi012 ceph-mon[53289]: pgmap v377: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-14T13:49:57.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:56 smithi012 ceph-mon[53289]: pgmap v378: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 228 B/s rd, 0 op/s 2024-04-14T13:49:57.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:56 smithi008 ceph-mon[67070]: pgmap v378: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 228 B/s rd, 0 op/s 2024-04-14T13:49:59.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:58 smithi012 ceph-mon[53289]: pgmap v379: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-14T13:49:59.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:58 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:59.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:49:58 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:59.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:58 smithi008 ceph-mon[67070]: pgmap v379: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-14T13:49:59.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:58 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:49:59.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:49:58 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:00.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:00 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:00.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:00 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:00.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:00 smithi008 ceph-mon[67070]: pgmap v380: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:00.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:00 smithi008 ceph-mon[67070]: overall HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T13:50:01.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:00 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:01.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:00 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:01.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:00 smithi012 ceph-mon[53289]: pgmap v380: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:01.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:00 smithi012 ceph-mon[53289]: overall HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T13:50:03.006 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:02 smithi008 ceph-mon[67070]: pgmap v381: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:03.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:02 smithi012 ceph-mon[53289]: pgmap v381: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:04.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:50:04.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:04.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:04.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:50:04.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:50:04.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:04.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:04.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T13:50:04.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:50:04.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:04.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T13:50:04.052 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T13:50:04.053 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T13:50:04.053 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T13:50:04.053 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T13:50:04.053 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T13:50:04.053 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T13:50:04.053 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-14T13:50:04.053 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T13:50:04.053 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-14T13:50:04.053 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T13:50:04.053 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-14T13:50:04.053 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T13:50:04.053 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-14T13:50:04.053 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T13:50:04.053 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T13:50:04.053 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-14T13:50:04.053 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T13:50:04.054 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-14T13:50:04.054 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T13:50:04.054 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-14T13:50:04.054 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T13:50:04.054 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T13:50:04.054 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T13:50:04.054 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T13:50:04.054 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T13:50:04.054 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T13:50:04.054 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T13:50:04.054 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T13:50:04.054 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T13:50:04.054 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T13:50:04.054 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:50:04.054 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:50:04.055 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:50:04.055 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:04.055 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:04.055 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:50:04.055 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:50:04.055 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:50:04.055 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:04.055 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:04.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:50:04.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:04.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:04.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:50:04.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:50:04.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:04.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:04.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T13:50:04.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:50:04.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:04.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:04.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T13:50:04.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T13:50:04.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T13:50:04.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T13:50:04.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T13:50:04.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T13:50:04.105 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T13:50:04.105 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-14T13:50:04.105 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T13:50:04.105 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-14T13:50:04.105 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T13:50:04.105 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-14T13:50:04.105 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T13:50:04.105 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-14T13:50:04.105 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T13:50:04.105 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T13:50:04.105 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-14T13:50:04.105 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T13:50:04.105 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-14T13:50:04.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T13:50:04.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-14T13:50:04.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T13:50:04.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T13:50:04.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T13:50:04.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T13:50:04.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T13:50:04.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T13:50:04.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T13:50:04.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T13:50:04.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T13:50:04.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T13:50:04.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:50:04.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:50:04.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:50:04.107 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:04.107 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:04.107 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:50:04.107 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:50:04.107 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:50:04.107 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:04.107 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:05.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:04 smithi012 ceph-mon[53289]: pgmap v382: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2024-04-14T13:50:05.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:04 smithi012 ceph-mon[53289]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T13:50:05.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:04 smithi012 ceph-mon[53289]: Upgrade: Finalizing container_image settings 2024-04-14T13:50:05.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:04 smithi012 ceph-mon[53289]: Upgrade: Complete! 2024-04-14T13:50:05.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:04 smithi012 ceph-mon[53289]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T13:50:05.051 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:04 smithi012 ceph-mon[53289]: Cluster is now healthy 2024-04-14T13:50:05.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:04 smithi008 ceph-mon[67070]: pgmap v382: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2024-04-14T13:50:05.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:04 smithi008 ceph-mon[67070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T13:50:05.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:04 smithi008 ceph-mon[67070]: Upgrade: Finalizing container_image settings 2024-04-14T13:50:05.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:04 smithi008 ceph-mon[67070]: Upgrade: Complete! 2024-04-14T13:50:05.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:04 smithi008 ceph-mon[67070]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T13:50:05.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:04 smithi008 ceph-mon[67070]: Cluster is now healthy 2024-04-14T13:50:07.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:06 smithi012 ceph-mon[53289]: pgmap v383: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 352 B/s rd, 0 op/s 2024-04-14T13:50:07.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:06 smithi008 ceph-mon[67070]: pgmap v383: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 352 B/s rd, 0 op/s 2024-04-14T13:50:09.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:08 smithi012 ceph-mon[53289]: pgmap v384: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2024-04-14T13:50:09.050 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:08 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:09.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:08 smithi008 ceph-mon[67070]: pgmap v384: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2024-04-14T13:50:09.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:08 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:11.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:10 smithi008 ceph-mon[67070]: pgmap v385: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2024-04-14T13:50:11.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:10 smithi012 ceph-mon[53289]: pgmap v385: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2024-04-14T13:50:13.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:12 smithi008 ceph-mon[67070]: pgmap v386: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2024-04-14T13:50:13.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:12 smithi012 ceph-mon[53289]: pgmap v386: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2024-04-14T13:50:15.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:14 smithi008 ceph-mon[67070]: pgmap v387: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2024-04-14T13:50:15.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:14 smithi012 ceph-mon[53289]: pgmap v387: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2024-04-14T13:50:16.779 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ps' 2024-04-14T13:50:17.096 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:16 smithi008 ceph-mon[67070]: pgmap v388: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:50:17.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:16 smithi012 ceph-mon[53289]: pgmap v388: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:50:18.054 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:17 smithi008 ceph-mon[67070]: from='client.34612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:50:18.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:17 smithi012 ceph-mon[53289]: from='client.34612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:50:18.755 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:50:18.755 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (40s) 20s ago 21m 16.6M - 0.25.0 c8568f914cd2 77146b15f08c 2024-04-14T13:50:18.755 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (8m) 20s ago 21m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 1e69eabdcc36 2024-04-14T13:50:18.755 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (8m) 71s ago 20m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 60c7a2992eb0 2024-04-14T13:50:18.755 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (25s) 20s ago 20m 53.0M - 9.4.12 2bacad6d85d8 07683ec85808 2024-04-14T13:50:18.755 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (2m) 20s ago 15m 35.2M - 19.0.0-2778-ga971fd5f b96c509eef84 50aacbc1bc98 2024-04-14T13:50:18.755 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (119s) 71s ago 15m 14.0M - 19.0.0-2778-ga971fd5f b96c509eef84 cdb17d8fe5ff 2024-04-14T13:50:18.755 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (11m) 20s ago 22m 524M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:50:18.755 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (10m) 71s ago 20m 454M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:50:18.755 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (9m) 20s ago 22m 63.0M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 673dd6f4c8d8 2024-04-14T13:50:18.755 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (9m) 71s ago 20m 48.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 d498dec29c4f 2024-04-14T13:50:18.755 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (105s) 20s ago 15m 72.6M - 5.7 b96c509eef84 0fc0feda12be 2024-04-14T13:50:18.756 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (99s) 71s ago 14m 68.2M - 5.7 b96c509eef84 56b616ccb6ed 2024-04-14T13:50:18.756 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (82s) 20s ago 20m 8094k - 1.5.0 0da6a335fe13 17213ca85724 2024-04-14T13:50:18.756 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (75s) 71s ago 20m 3363k - 1.5.0 0da6a335fe13 df7b43405919 2024-04-14T13:50:18.756 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (7m) 20s ago 19m 394M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 29997ccc66ac 2024-04-14T13:50:18.756 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (6m) 20s ago 19m 363M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 97df88f10bd6 2024-04-14T13:50:18.756 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (5m) 20s ago 18m 326M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 9325b527c16c 2024-04-14T13:50:18.756 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (5m) 20s ago 18m 430M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 6868b5076d61 2024-04-14T13:50:18.756 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (4m) 71s ago 18m 366M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 178f9462f5ba 2024-04-14T13:50:18.756 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (4m) 71s ago 17m 317M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 25e416621d40 2024-04-14T13:50:18.756 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (3m) 71s ago 17m 293M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 97f0b691cf87 2024-04-14T13:50:18.756 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (2m) 71s ago 16m 289M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 7e4e731dec31 2024-04-14T13:50:18.756 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (58s) 20s ago 19m 53.6M - 2.43.0 a07b618ecd1d 492f8cdc3836 2024-04-14T13:50:19.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:18 smithi008 ceph-mon[67070]: pgmap v389: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:19.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:18 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:19.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:18 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:50:19.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:18 smithi012 ceph-mon[53289]: pgmap v389: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:19.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:18 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:50:19.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:18 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:50:19.412 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions' 2024-04-14T13:50:19.977 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:19 smithi008 ceph-mon[67070]: from='client.34616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:50:20.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:19 smithi012 ceph-mon[53289]: from='client.34616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:50:20.927 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:20 smithi008 ceph-mon[67070]: pgmap v390: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:21.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:20 smithi012 ceph-mon[53289]: pgmap v390: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:21.495 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:50:21.495 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:50:21.495 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:50:21.495 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:50:21.495 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:50:21.495 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:50:21.495 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:50:21.495 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:50:21.495 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 8 2024-04-14T13:50:21.495 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:50:21.495 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:50:21.495 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:50:21.495 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:50:21.495 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:50:21.496 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 14 2024-04-14T13:50:21.496 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:50:21.496 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:50:22.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:21 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/4286910782' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:22.243 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-14T13:50:22.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:21 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/4286910782' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:50:23.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:22 smithi008 ceph-mon[67070]: pgmap v391: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:23.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:22 smithi012 ceph-mon[53289]: pgmap v391: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:24.006 INFO:teuthology.orchestra.run.smithi008.stdout:wait for servicemap items w/ changing names to refresh 2024-04-14T13:50:24.682 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'sleep 60' 2024-04-14T13:50:24.999 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:24 smithi008 ceph-mon[67070]: pgmap v392: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:25.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:24 smithi012 ceph-mon[53289]: pgmap v392: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:27.102 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:26 smithi012 ceph-mon[53289]: pgmap v393: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:50:27.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:26 smithi008 ceph-mon[67070]: pgmap v393: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:50:29.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:28 smithi008 ceph-mon[67070]: pgmap v394: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:29.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:28 smithi012 ceph-mon[53289]: pgmap v394: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:31.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:30 smithi008 ceph-mon[67070]: pgmap v395: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:31.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:30 smithi012 ceph-mon[53289]: pgmap v395: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:33.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:32 smithi008 ceph-mon[67070]: pgmap v396: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:33.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:32 smithi012 ceph-mon[53289]: pgmap v396: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:34.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:33 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:50:34.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:33 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:50:35.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:34 smithi008 ceph-mon[67070]: pgmap v397: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:35.302 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:34 smithi012 ceph-mon[53289]: pgmap v397: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:37.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:36 smithi012 ceph-mon[53289]: pgmap v398: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:50:37.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:36 smithi008 ceph-mon[67070]: pgmap v398: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:50:39.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:38 smithi008 ceph-mon[67070]: pgmap v399: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:39.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:38 smithi012 ceph-mon[53289]: pgmap v399: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:41.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:40 smithi012 ceph-mon[53289]: pgmap v400: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:41.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:40 smithi008 ceph-mon[67070]: pgmap v400: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:43.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:42 smithi012 ceph-mon[53289]: pgmap v401: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:43.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:42 smithi008 ceph-mon[67070]: pgmap v401: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:45.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:44 smithi012 ceph-mon[53289]: pgmap v402: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:45.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:44 smithi008 ceph-mon[67070]: pgmap v402: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:47.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:46 smithi012 ceph-mon[53289]: pgmap v403: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:50:47.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:46 smithi008 ceph-mon[67070]: pgmap v403: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:50:49.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:48 smithi008 ceph-mon[67070]: pgmap v404: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:49.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:50:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:48 smithi012 ceph-mon[53289]: pgmap v404: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:50:51.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:50 smithi012 ceph-mon[53289]: pgmap v405: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:51.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:50 smithi008 ceph-mon[67070]: pgmap v405: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:53.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:52 smithi012 ceph-mon[53289]: pgmap v406: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:53.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:52 smithi008 ceph-mon[67070]: pgmap v406: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:55.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:54 smithi012 ceph-mon[53289]: pgmap v407: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:55.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:54 smithi008 ceph-mon[67070]: pgmap v407: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:57.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:56 smithi012 ceph-mon[53289]: pgmap v408: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:50:57.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:56 smithi008 ceph-mon[67070]: pgmap v408: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:50:59.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:50:58 smithi008 ceph-mon[67070]: pgmap v409: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:50:59.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:50:58 smithi012 ceph-mon[53289]: pgmap v409: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:01.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:00 smithi012 ceph-mon[53289]: pgmap v410: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:01.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:00 smithi008 ceph-mon[67070]: pgmap v410: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:03.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:02 smithi012 ceph-mon[53289]: pgmap v411: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:03.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:02 smithi008 ceph-mon[67070]: pgmap v411: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:04.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:51:04.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:03 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:51:04.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:51:04.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:03 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:51:05.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:04 smithi012 ceph-mon[53289]: pgmap v412: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:05.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:04 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:51:05.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:04 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:51:05.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:04 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:51:05.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:04 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:51:05.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:04 smithi008 ceph-mon[67070]: pgmap v412: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:05.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:04 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:51:05.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:04 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:51:05.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:04 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:51:05.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:04 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' 2024-04-14T13:51:07.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:06 smithi012 ceph-mon[53289]: pgmap v413: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:51:07.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:06 smithi008 ceph-mon[67070]: pgmap v413: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:51:09.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:08 smithi008 ceph-mon[67070]: pgmap v414: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:09.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:08 smithi012 ceph-mon[53289]: pgmap v414: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:11.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:10 smithi012 ceph-mon[53289]: pgmap v415: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:11.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:10 smithi008 ceph-mon[67070]: pgmap v415: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:13.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:12 smithi012 ceph-mon[53289]: pgmap v416: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:13.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:12 smithi008 ceph-mon[67070]: pgmap v416: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:15.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:14 smithi012 ceph-mon[53289]: pgmap v417: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:15.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:14 smithi008 ceph-mon[67070]: pgmap v417: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:17.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:16 smithi012 ceph-mon[53289]: pgmap v418: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:51:17.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:16 smithi008 ceph-mon[67070]: pgmap v418: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:51:19.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:18 smithi012 ceph-mon[53289]: pgmap v419: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:19.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:18 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:51:19.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:18 smithi008 ceph-mon[67070]: pgmap v419: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:19.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:18 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:51:21.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:20 smithi012 ceph-mon[53289]: pgmap v420: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:21.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:20 smithi008 ceph-mon[67070]: pgmap v420: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:23.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:22 smithi012 ceph-mon[53289]: pgmap v421: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:23.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:22 smithi008 ceph-mon[67070]: pgmap v421: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:25.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:24 smithi012 ceph-mon[53289]: pgmap v422: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:25.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:24 smithi008 ceph-mon[67070]: pgmap v422: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:26.956 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:26 smithi008 ceph-mon[67070]: pgmap v423: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:51:26.985 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ps' 2024-04-14T13:51:27.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:26 smithi012 ceph-mon[53289]: pgmap v423: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:51:28.961 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T13:51:28.962 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (111s) 90s ago 22m 16.6M - 0.25.0 c8568f914cd2 77146b15f08c 2024-04-14T13:51:28.962 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (9m) 90s ago 22m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 1e69eabdcc36 2024-04-14T13:51:28.962 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi012 smithi012 running (9m) 2m ago 21m 6660k - 19.0.0-2778-ga971fd5f b96c509eef84 60c7a2992eb0 2024-04-14T13:51:28.962 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (96s) 90s ago 22m 53.0M - 9.4.12 2bacad6d85d8 07683ec85808 2024-04-14T13:51:28.962 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wqbvfi smithi008 running (3m) 90s ago 16m 35.2M - 19.0.0-2778-ga971fd5f b96c509eef84 50aacbc1bc98 2024-04-14T13:51:28.962 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi012.vjovzp smithi012 running (3m) 2m ago 16m 14.0M - 19.0.0-2778-ga971fd5f b96c509eef84 cdb17d8fe5ff 2024-04-14T13:51:28.962 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.yemlhr smithi008 *:8443,9283,8765 running (12m) 90s ago 23m 524M - 19.0.0-2778-ga971fd5f b96c509eef84 66e433d36a6b 2024-04-14T13:51:28.962 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi012.ksvnxm smithi012 *:8443,9283,8765 running (11m) 2m ago 21m 454M - 19.0.0-2778-ga971fd5f b96c509eef84 d5c28122977c 2024-04-14T13:51:28.962 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (10m) 90s ago 23m 63.0M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 673dd6f4c8d8 2024-04-14T13:51:28.962 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi012 smithi012 running (10m) 2m ago 21m 48.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 d498dec29c4f 2024-04-14T13:51:28.962 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.vefdqq smithi008 *:2049 running (2m) 90s ago 16m 72.6M - 5.7 b96c509eef84 0fc0feda12be 2024-04-14T13:51:28.962 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi012.meswmn smithi012 *:2049 running (2m) 2m ago 16m 68.2M - 5.7 b96c509eef84 56b616ccb6ed 2024-04-14T13:51:28.962 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (2m) 90s ago 22m 8094k - 1.5.0 0da6a335fe13 17213ca85724 2024-04-14T13:51:28.963 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi012 smithi012 *:9100 running (2m) 2m ago 21m 3363k - 1.5.0 0da6a335fe13 df7b43405919 2024-04-14T13:51:28.963 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (8m) 90s ago 20m 394M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 29997ccc66ac 2024-04-14T13:51:28.963 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (7m) 90s ago 20m 363M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 97df88f10bd6 2024-04-14T13:51:28.963 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (7m) 90s ago 20m 326M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 9325b527c16c 2024-04-14T13:51:28.963 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (6m) 90s ago 19m 430M 1937M 19.0.0-2778-ga971fd5f b96c509eef84 6868b5076d61 2024-04-14T13:51:28.963 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi012 running (5m) 2m ago 19m 366M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 178f9462f5ba 2024-04-14T13:51:28.963 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi012 running (5m) 2m ago 18m 317M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 25e416621d40 2024-04-14T13:51:28.963 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi012 running (4m) 2m ago 18m 293M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 97f0b691cf87 2024-04-14T13:51:28.963 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi012 running (3m) 2m ago 17m 289M 2705M 19.0.0-2778-ga971fd5f b96c509eef84 7e4e731dec31 2024-04-14T13:51:28.963 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (2m) 90s ago 21m 53.6M - 2.43.0 a07b618ecd1d 492f8cdc3836 2024-04-14T13:51:29.255 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:28 smithi008 ceph-mon[67070]: pgmap v424: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:29.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:28 smithi012 ceph-mon[53289]: pgmap v424: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:29.579 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade status' 2024-04-14T13:51:30.166 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:29 smithi008 ceph-mon[67070]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:51:30.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:29 smithi012 ceph-mon[53289]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:51:31.232 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:30 smithi008 ceph-mon[67070]: pgmap v425: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:31.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:30 smithi012 ceph-mon[53289]: pgmap v425: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:31.574 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:51:31.574 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": null, 2024-04-14T13:51:31.574 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": false, 2024-04-14T13:51:31.574 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "", 2024-04-14T13:51:31.574 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-04-14T13:51:31.574 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": null, 2024-04-14T13:51:31.574 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "", 2024-04-14T13:51:31.574 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-14T13:51:31.574 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:51:32.205 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph health detail' 2024-04-14T13:51:33.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:32 smithi008 ceph-mon[67070]: pgmap v426: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:33.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:32 smithi008 ceph-mon[67070]: from='client.34628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:51:33.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:32 smithi012 ceph-mon[53289]: pgmap v426: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:33.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:32 smithi012 ceph-mon[53289]: from='client.34628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:51:34.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:33 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:51:34.280 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-04-14T13:51:34.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:33 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:51:34.877 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions' 2024-04-14T13:51:35.179 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:34 smithi008 ceph-mon[67070]: pgmap v427: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:35.179 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:34 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/3913242804' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:51:35.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:34 smithi012 ceph-mon[53289]: pgmap v427: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:35.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:34 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/3913242804' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T13:51:36.870 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-14T13:51:36.870 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-14T13:51:36.870 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:51:36.870 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:51:36.870 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-14T13:51:36.870 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:51:36.871 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:51:36.871 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-14T13:51:36.871 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 8 2024-04-14T13:51:36.871 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:51:36.871 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-14T13:51:36.871 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T13:51:36.871 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:51:36.871 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-14T13:51:36.871 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 14 2024-04-14T13:51:36.871 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:51:36.871 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-14T13:51:37.184 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:36 smithi008 ceph-mon[67070]: pgmap v428: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:51:37.184 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:36 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/1972232461' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:51:37.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:36 smithi012 ceph-mon[53289]: pgmap v428: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:51:37.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:36 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/1972232461' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:51:37.511 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-14T13:51:39.023 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:38 smithi008 ceph-mon[67070]: pgmap v429: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:39.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:38 smithi012 ceph-mon[53289]: pgmap v429: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:39.598 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-14T13:51:40.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:39 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/3848684757' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:51:40.198 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-14T13:51:40.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:39 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/3848684757' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:51:41.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:40 smithi008 ceph-mon[67070]: pgmap v430: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:41.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:40 smithi012 ceph-mon[53289]: pgmap v430: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:42.251 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)" 2024-04-14T13:51:42.905 INFO:teuthology.task.sequential:In sequential, running task cephadm.wait_for_service... 2024-04-14T13:51:42.906 INFO:tasks.cephadm:Waiting for ceph service nfs.foo to start (timeout 300)... 2024-04-14T13:51:42.906 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- ceph orch ls -f json 2024-04-14T13:51:43.218 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:42 smithi008 ceph-mon[67070]: pgmap v431: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:43.218 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:42 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/3788517317' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:51:43.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:42 smithi012 ceph-mon[53289]: pgmap v431: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:43.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:42 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/3788517317' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T13:51:44.863 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-14T13:51:44.863 INFO:teuthology.orchestra.run.smithi008.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-14T13:28:24.640841Z", "last_refresh": "2024-04-14T13:49:58.558138Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-14T13:28:19.222610Z", "last_refresh": "2024-04-14T13:49:07.192845Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "spec": {"anonymous_access": true, "protocol": "https"}, "status": {"created": "2024-04-14T13:28:22.183609Z", "last_refresh": "2024-04-14T13:49:58.558301Z", "ports": [3000], "running": 1, "size": 1}}, {"placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-14T13:34:52.565217Z", "last_refresh": "2024-04-14T13:49:07.193511Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-14T13:28:17.214703Z", "last_refresh": "2024-04-14T13:49:07.192950Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi008:172.21.15.8=smithi008", "smithi012:172.21.15.12=smithi012"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-14T13:29:07.440988Z", "last_refresh": "2024-04-14T13:49:07.193032Z", "running": 2, "size": 2}}, {"events": ["2024-04-14T13:51:04.501491Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 2049}, "status": {"created": "2024-04-14T13:35:09.933608Z", "last_refresh": "2024-04-14T13:49:07.193589Z", "ports": [2049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-14T13:28:23.319722Z", "last_refresh": "2024-04-14T13:49:07.193111Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53", "container_image_name": "quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608", "last_refresh": "2024-04-14T13:49:07.193200Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-14T13:28:20.882885Z", "last_refresh": "2024-04-14T13:49:58.558455Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-14T13:51:45.164 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:44 smithi008 ceph-mon[67070]: pgmap v432: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:45.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:44 smithi012 ceph-mon[53289]: pgmap v432: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:45.463 INFO:tasks.cephadm:nfs.foo has 2/2 2024-04-14T13:51:45.464 DEBUG:teuthology.parallel:result is None 2024-04-14T13:51:45.464 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-14T13:51:45.475 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-14T13:51:45.475 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'umount /mnt/foo' 2024-04-14T13:51:45.503 INFO:teuthology.orchestra.run.smithi008.stderr:+ umount /mnt/foo 2024-04-14T13:51:45.515 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-14T13:51:45.526 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-14T13:51:45.526 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- bash -c 'ceph nfs cluster ls | grep foo' 2024-04-14T13:51:46.135 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:45 smithi008 ceph-mon[67070]: from='client.34648 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:51:46.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:45 smithi012 ceph-mon[53289]: from='client.34648 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:51:47.013 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:46 smithi008 ceph-mon[67070]: pgmap v433: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:51:47.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:46 smithi012 ceph-mon[53289]: pgmap v433: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T13:51:47.585 INFO:teuthology.orchestra.run.smithi008.stdout: "foo" 2024-04-14T13:51:48.231 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- bash -c 'ceph nfs export ls foo --detailed' 2024-04-14T13:51:49.207 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:48 smithi008 ceph-mon[67070]: pgmap v434: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:49.207 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:48 smithi008 ceph-mon[67070]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:51:49.208 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:48 smithi008 ceph-mon[67070]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:51:49.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:48 smithi012 ceph-mon[53289]: pgmap v434: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:48 smithi012 ceph-mon[53289]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:51:49.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:48 smithi012 ceph-mon[53289]: from='mgr.34100 172.21.15.8:0/4011194411' entity='mgr.smithi008.yemlhr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T13:51:50.253 INFO:teuthology.orchestra.run.smithi008.stdout:[ 2024-04-14T13:51:50.253 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-14T13:51:50.253 INFO:teuthology.orchestra.run.smithi008.stdout: "access_type": "RW", 2024-04-14T13:51:50.253 INFO:teuthology.orchestra.run.smithi008.stdout: "clients": [], 2024-04-14T13:51:50.253 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_id": "foo", 2024-04-14T13:51:50.253 INFO:teuthology.orchestra.run.smithi008.stdout: "export_id": 1, 2024-04-14T13:51:50.253 INFO:teuthology.orchestra.run.smithi008.stdout: "fsal": { 2024-04-14T13:51:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: "fs_name": "foofs", 2024-04-14T13:51:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: "name": "CEPH", 2024-04-14T13:51:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: "user_id": "nfs.foo.1" 2024-04-14T13:51:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-14T13:51:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: "path": "/", 2024-04-14T13:51:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: "protocols": [ 2024-04-14T13:51:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: 4 2024-04-14T13:51:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-14T13:51:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: "pseudo": "/fake", 2024-04-14T13:51:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: "security_label": true, 2024-04-14T13:51:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: "squash": "none", 2024-04-14T13:51:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: "transports": [ 2024-04-14T13:51:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: "TCP" 2024-04-14T13:51:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-14T13:51:50.254 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-14T13:51:50.255 INFO:teuthology.orchestra.run.smithi008.stdout:] 2024-04-14T13:51:50.865 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- bash -c 'rados -p .nfs --all ls -' 2024-04-14T13:51:51.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:50 smithi008 ceph-mon[67070]: pgmap v435: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:51.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:50 smithi012 ceph-mon[53289]: pgmap v435: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:52.258 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:51 smithi008 ceph-mon[67070]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:51:52.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:51 smithi012 ceph-mon[53289]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:51:52.568 INFO:teuthology.orchestra.run.smithi008.stdout:foo rec-0000000000000004:nfs.foo.1 2024-04-14T13:51:52.568 INFO:teuthology.orchestra.run.smithi008.stdout:foo grace 2024-04-14T13:51:52.568 INFO:teuthology.orchestra.run.smithi008.stdout:foo rec-0000000000000004:nfs.foo.0 2024-04-14T13:51:52.568 INFO:teuthology.orchestra.run.smithi008.stdout:foo conf-nfs.foo 2024-04-14T13:51:52.569 INFO:teuthology.orchestra.run.smithi008.stdout:foo export-1 2024-04-14T13:51:53.264 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:52 smithi008 ceph-mon[67070]: pgmap v436: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:53.291 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-14T13:51:53.300 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-14T13:51:53.301 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 -- bash -c 'set -ex 2024-04-14T13:51:53.301 DEBUG:teuthology.orchestra.run.smithi008:> [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] 2024-04-14T13:51:53.301 DEBUG:teuthology.orchestra.run.smithi008:> ' 2024-04-14T13:51:53.301 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:52 smithi012 ceph-mon[53289]: pgmap v436: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T13:51:54.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:53 smithi012 ceph-mon[53289]: mgrmap e43: smithi008.yemlhr(active, since 11m), standbys: smithi012.ksvnxm 2024-04-14T13:51:54.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:53 smithi008 ceph-mon[67070]: mgrmap e43: smithi008.yemlhr(active, since 11m), standbys: smithi012.ksvnxm 2024-04-14T13:51:54.998 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:54 smithi008 ceph-mon[67070]: pgmap v437: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-14T13:51:55.092 INFO:teuthology.orchestra.run.smithi008.stderr:++ ceph config get mgr mgr/cephadm/migration_current 2024-04-14T13:51:55.300 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:54 smithi012 ceph-mon[53289]: pgmap v437: 97 pgs: 97 active+clean; 227 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-14T13:51:55.330 INFO:teuthology.orchestra.run.smithi008.stderr:+ [[ 6 -gt 2 ]] 2024-04-14T13:51:55.894 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-14T13:51:55.904 INFO:tasks.cephadm:Teardown begin 2024-04-14T13:51:55.905 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-14T13:51:55.934 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-14T13:51:55.969 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-14T13:51:55.969 DEBUG:teuthology.orchestra.run.smithi008:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-14T13:51:55.989 DEBUG:teuthology.orchestra.run.smithi012:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-14T13:51:56.026 INFO:tasks.cephadm:Stopping all daemons... 2024-04-14T13:51:56.027 INFO:tasks.cephadm.mon.smithi008:Stopping mon.smithi008... 2024-04-14T13:51:56.027 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@mon.smithi008 2024-04-14T13:51:56.299 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:55 smithi012 ceph-mon[53289]: from='client.? 172.21.15.8:0/2086799902' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-04-14T13:51:56.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:55 smithi008 ceph-mon[67070]: from='client.? 172.21.15.8:0/2086799902' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-04-14T13:51:56.351 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:56 smithi008 systemd[1]: Stopping Ceph mon.smithi008 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:51:56.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:56 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008[67065]: 2024-04-14T13:51:56.347+0000 7fc9a1d60700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi008 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:51:56.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:56 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008[67065]: 2024-04-14T13:51:56.347+0000 7fc9a1d60700 -1 mon.smithi008@0(leader) e3 *** Got Signal Terminated *** 2024-04-14T13:51:56.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:56 smithi008 podman[112063]: 2024-04-14 13:51:56.37937795 +0000 UTC m=+0.166584506 container died 673dd6f4c8d803e16c0354275d3ad8ee2ca56b3d1f2920791f3265d81b95724e (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:51:57.002 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:56 smithi008 podman[112063]: 2024-04-14 13:51:56.748744957 +0000 UTC m=+0.535951515 container cleanup 673dd6f4c8d803e16c0354275d3ad8ee2ca56b3d1f2920791f3265d81b95724e (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-14T13:51:57.002 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 14 13:51:56 smithi008 bash[112063]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi008 2024-04-14T13:51:57.167 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@mon.smithi008.service' 2024-04-14T13:51:57.223 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T13:51:57.223 INFO:tasks.cephadm.mon.smithi008:Stopped mon.smithi008 2024-04-14T13:51:57.223 INFO:tasks.cephadm.mon.smithi012:Stopping mon.smithi012... 2024-04-14T13:51:57.223 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@mon.smithi012 2024-04-14T13:51:57.538 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:57 smithi012 systemd[1]: Stopping Ceph mon.smithi012 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:51:57.790 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:57 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi012[53285]: 2024-04-14T13:51:57.534+0000 7f2d4777f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi012 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:51:57.790 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:57 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi012[53285]: 2024-04-14T13:51:57.534+0000 7f2d4777f700 -1 mon.smithi012@1(peon) e3 *** Got Signal Terminated *** 2024-04-14T13:51:57.790 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:57 smithi012 podman[78125]: 2024-04-14 13:51:57.566538376 +0000 UTC m=+0.147533356 container died d498dec29c4f3f1cade97459209d271cb28a9089501bbaca04826f66f4e2cf42 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi012, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-a971fd5, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2) 2024-04-14T13:51:58.357 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:58 smithi012 podman[78125]: 2024-04-14 13:51:58.086625011 +0000 UTC m=+0.667619980 container cleanup d498dec29c4f3f1cade97459209d271cb28a9089501bbaca04826f66f4e2cf42 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi012, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:51:58.357 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:58 smithi012 bash[78125]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi012 2024-04-14T13:51:58.653 INFO:journalctl@ceph.mon.smithi012.smithi012.stdout:Apr 14 13:51:58 smithi012 podman[78139]: 2024-04-14 13:51:58.35497086 +0000 UTC m=+0.781253603 container remove d498dec29c4f3f1cade97459209d271cb28a9089501bbaca04826f66f4e2cf42 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-mon-smithi012, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T13:51:58.677 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@mon.smithi012.service' 2024-04-14T13:51:58.728 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T13:51:58.728 INFO:tasks.cephadm.mon.smithi012:Stopped mon.smithi012 2024-04-14T13:51:58.728 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-14T13:51:58.728 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.0 2024-04-14T13:51:59.038 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:51:58 smithi008 systemd[1]: Stopping Ceph osd.0 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:51:59.352 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:51:59 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0[73142]: 2024-04-14T13:51:59.035+0000 7f4b36fe5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:51:59.352 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:51:59 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0[73142]: 2024-04-14T13:51:59.035+0000 7f4b36fe5700 -1 osd.0 154 *** Got signal Terminated *** 2024-04-14T13:51:59.352 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:51:59 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0[73142]: 2024-04-14T13:51:59.035+0000 7f4b36fe5700 -1 osd.0 154 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T13:52:04.947 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:52:04 smithi008 podman[112194]: 2024-04-14 13:52:04.603911718 +0000 UTC m=+5.679549015 container died 29997ccc66acd144ae05a4a5790218e0c363d45c0e56d235ce154b72985a5743 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-14T13:52:05.256 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:52:05 smithi008 podman[112194]: 2024-04-14 13:52:05.02793358 +0000 UTC m=+6.103570874 container cleanup 29997ccc66acd144ae05a4a5790218e0c363d45c0e56d235ce154b72985a5743 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T13:52:05.257 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:52:05 smithi008 bash[112194]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0 2024-04-14T13:52:05.602 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:52:05 smithi008 podman[112245]: 2024-04-14 13:52:05.254803231 +0000 UTC m=+0.651486295 container remove 29997ccc66acd144ae05a4a5790218e0c363d45c0e56d235ce154b72985a5743 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True) 2024-04-14T13:52:06.018 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:52:05 smithi008 podman[112338]: 2024-04-14 13:52:05.603655699 +0000 UTC m=+0.018911376 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:52:06.018 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:52:05 smithi008 podman[112338]: 2024-04-14 13:52:05.7140467 +0000 UTC m=+0.129302371 container create 5a30be7bdfa95ecc880462816c109ab23b6fe1dfe7365bbff1b7a4ecf7714fbd (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, io.buildah.version=1.33.5) 2024-04-14T13:52:06.019 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:52:05 smithi008 podman[112338]: 2024-04-14 13:52:05.908373654 +0000 UTC m=+0.323629325 container init 5a30be7bdfa95ecc880462816c109ab23b6fe1dfe7365bbff1b7a4ecf7714fbd (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-14T13:52:06.019 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:52:05 smithi008 podman[112338]: 2024-04-14 13:52:05.911809993 +0000 UTC m=+0.327065673 container start 5a30be7bdfa95ecc880462816c109ab23b6fe1dfe7365bbff1b7a4ecf7714fbd (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-14T13:52:06.019 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:52:06 smithi008 podman[112338]: 2024-04-14 13:52:06.016629509 +0000 UTC m=+0.431885182 container attach 5a30be7bdfa95ecc880462816c109ab23b6fe1dfe7365bbff1b7a4ecf7714fbd (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T13:52:06.352 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 14 13:52:06 smithi008 podman[112338]: 2024-04-14 13:52:06.2108626 +0000 UTC m=+0.626118274 container died 5a30be7bdfa95ecc880462816c109ab23b6fe1dfe7365bbff1b7a4ecf7714fbd (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-a971fd5, GIT_CLEAN=True) 2024-04-14T13:52:06.819 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.0.service' 2024-04-14T13:52:06.864 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T13:52:06.864 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-14T13:52:06.864 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-14T13:52:06.865 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.1 2024-04-14T13:52:07.102 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:52:06 smithi008 systemd[1]: Stopping Ceph osd.1 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:52:07.471 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:52:07 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1[75673]: 2024-04-14T13:52:07.158+0000 7f3b861d4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:52:07.471 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:52:07 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1[75673]: 2024-04-14T13:52:07.158+0000 7f3b861d4700 -1 osd.1 154 *** Got signal Terminated *** 2024-04-14T13:52:07.471 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:52:07 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1[75673]: 2024-04-14T13:52:07.158+0000 7f3b861d4700 -1 osd.1 154 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T13:52:12.854 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:52:12 smithi008 podman[112443]: 2024-04-14 13:52:12.594148555 +0000 UTC m=+5.544112356 container died 97df88f10bd6aa144a1bda874a67d40ea31cf24c89f592ec5346f23e1fb72673 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T13:52:13.212 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:52:12 smithi008 podman[112443]: 2024-04-14 13:52:12.984288016 +0000 UTC m=+5.934251820 container cleanup 97df88f10bd6aa144a1bda874a67d40ea31cf24c89f592ec5346f23e1fb72673 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:52:13.212 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:52:12 smithi008 bash[112443]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1 2024-04-14T13:52:13.602 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:52:13 smithi008 podman[112460]: 2024-04-14 13:52:13.210568782 +0000 UTC m=+0.617558529 container remove 97df88f10bd6aa144a1bda874a67d40ea31cf24c89f592ec5346f23e1fb72673 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, io.buildah.version=1.33.5) 2024-04-14T13:52:13.907 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:52:13 smithi008 podman[112552]: 2024-04-14 13:52:13.51419646 +0000 UTC m=+0.018461522 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:52:13.908 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:52:13 smithi008 podman[112552]: 2024-04-14 13:52:13.617607256 +0000 UTC m=+0.121872306 container create b0e6b46d1259e64ae37e6cc8b581be80e7ca4dad84122b1429a4d08c98a1e8bc (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, ceph=True) 2024-04-14T13:52:13.908 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:52:13 smithi008 podman[112552]: 2024-04-14 13:52:13.823578285 +0000 UTC m=+0.327843343 container init b0e6b46d1259e64ae37e6cc8b581be80e7ca4dad84122b1429a4d08c98a1e8bc (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T13:52:13.908 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:52:13 smithi008 podman[112552]: 2024-04-14 13:52:13.827048363 +0000 UTC m=+0.331313422 container start b0e6b46d1259e64ae37e6cc8b581be80e7ca4dad84122b1429a4d08c98a1e8bc (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-deactivate, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403) 2024-04-14T13:52:13.908 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:52:13 smithi008 podman[112552]: 2024-04-14 13:52:13.905984506 +0000 UTC m=+0.410249562 container attach b0e6b46d1259e64ae37e6cc8b581be80e7ca4dad84122b1429a4d08c98a1e8bc (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-14T13:52:14.352 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 14 13:52:14 smithi008 podman[112552]: 2024-04-14 13:52:14.12815958 +0000 UTC m=+0.632424631 container died b0e6b46d1259e64ae37e6cc8b581be80e7ca4dad84122b1429a4d08c98a1e8bc (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-1-deactivate, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-14T13:52:14.718 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.1.service' 2024-04-14T13:52:14.763 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T13:52:14.763 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-14T13:52:14.763 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-14T13:52:14.763 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.2 2024-04-14T13:52:15.073 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:52:14 smithi008 systemd[1]: Stopping Ceph osd.2 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:52:15.352 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:52:15 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2[78375]: 2024-04-14T13:52:15.070+0000 7f7b99e84700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:52:15.352 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:52:15 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2[78375]: 2024-04-14T13:52:15.070+0000 7f7b99e84700 -1 osd.2 154 *** Got signal Terminated *** 2024-04-14T13:52:15.352 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:52:15 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2[78375]: 2024-04-14T13:52:15.070+0000 7f7b99e84700 -1 osd.2 154 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T13:52:20.924 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:52:20 smithi008 podman[112659]: 2024-04-14 13:52:20.65561003 +0000 UTC m=+5.707446219 container died 9325b527c16c31beb9a724107e170aabcb3f4fc4743f8f604fcf465c3d19a5e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T13:52:21.199 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:52:20 smithi008 podman[112659]: 2024-04-14 13:52:20.995997444 +0000 UTC m=+6.047833624 container cleanup 9325b527c16c31beb9a724107e170aabcb3f4fc4743f8f604fcf465c3d19a5e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-04-14T13:52:21.199 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:52:20 smithi008 bash[112659]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2 2024-04-14T13:52:21.602 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:52:21 smithi008 podman[112676]: 2024-04-14 13:52:21.197609662 +0000 UTC m=+0.540815238 container remove 9325b527c16c31beb9a724107e170aabcb3f4fc4743f8f604fcf465c3d19a5e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T13:52:21.911 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:52:21 smithi008 podman[112769]: 2024-04-14 13:52:21.535681641 +0000 UTC m=+0.018976894 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:52:21.911 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:52:21 smithi008 podman[112769]: 2024-04-14 13:52:21.640249485 +0000 UTC m=+0.123544757 container create 06333d498365f6c82a8595dac0b3babfc848ac18ad5db5e2f023fe2d2b202e0e (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-04-14T13:52:21.911 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:52:21 smithi008 podman[112769]: 2024-04-14 13:52:21.83423024 +0000 UTC m=+0.317525471 container init 06333d498365f6c82a8595dac0b3babfc848ac18ad5db5e2f023fe2d2b202e0e (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:52:21.911 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:52:21 smithi008 podman[112769]: 2024-04-14 13:52:21.837640092 +0000 UTC m=+0.320935323 container start 06333d498365f6c82a8595dac0b3babfc848ac18ad5db5e2f023fe2d2b202e0e (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-04-14T13:52:21.911 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:52:21 smithi008 podman[112769]: 2024-04-14 13:52:21.909251702 +0000 UTC m=+0.392546936 container attach 06333d498365f6c82a8595dac0b3babfc848ac18ad5db5e2f023fe2d2b202e0e (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-14T13:52:22.352 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 14 13:52:22 smithi008 podman[112769]: 2024-04-14 13:52:22.135521185 +0000 UTC m=+0.618816417 container died 06333d498365f6c82a8595dac0b3babfc848ac18ad5db5e2f023fe2d2b202e0e (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-04-14T13:52:22.701 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.2.service' 2024-04-14T13:52:22.745 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T13:52:22.746 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-14T13:52:22.746 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-14T13:52:22.746 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.3 2024-04-14T13:52:23.077 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:52:22 smithi008 systemd[1]: Stopping Ceph osd.3 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:52:23.351 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:52:23 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3[80886]: 2024-04-14T13:52:23.074+0000 7f36966ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:52:23.352 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:52:23 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3[80886]: 2024-04-14T13:52:23.074+0000 7f36966ed700 -1 osd.3 154 *** Got signal Terminated *** 2024-04-14T13:52:23.352 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:52:23 smithi008 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3[80886]: 2024-04-14T13:52:23.074+0000 7f36966ed700 -1 osd.3 154 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T13:52:26.050 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:25 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:25.665+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:27.050 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:26 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:26.850+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:27.050 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:26 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:26.842+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:02.963468+0000 front 2024-04-14T13:52:02.963389+0000 (oldest deadline 2024-04-14T13:52:26.463537+0000) 2024-04-14T13:52:27.051 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:26 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:26.617+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:27.835 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:27 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:27.580+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:28.300 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:27 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:27.871+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:02.963468+0000 front 2024-04-14T13:52:02.963389+0000 (oldest deadline 2024-04-14T13:52:26.463537+0000) 2024-04-14T13:52:28.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:27 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:27.833+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:28.831 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:28 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:28.577+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:28.852 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:52:28 smithi008 podman[112905]: 2024-04-14 13:52:28.58790751 +0000 UTC m=+5.617443449 container died 6868b5076d616a154d9e059abb78bb82367a53545d2bf983737ef791aaf44597 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-14T13:52:29.200 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:52:28 smithi008 podman[112905]: 2024-04-14 13:52:28.966095682 +0000 UTC m=+5.995631603 container cleanup 6868b5076d616a154d9e059abb78bb82367a53545d2bf983737ef791aaf44597 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T13:52:29.201 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:52:28 smithi008 bash[112905]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3 2024-04-14T13:52:29.299 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:28 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:28.866+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:02.963468+0000 front 2024-04-14T13:52:02.963389+0000 (oldest deadline 2024-04-14T13:52:26.463537+0000) 2024-04-14T13:52:29.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:28 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:28.829+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:29.559 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:52:29 smithi008 podman[112923]: 2024-04-14 13:52:29.225879129 +0000 UTC m=+0.638801012 container remove 6868b5076d616a154d9e059abb78bb82367a53545d2bf983737ef791aaf44597 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:52:29.818 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:29 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:29.562+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:29.852 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:52:29 smithi008 podman[113026]: 2024-04-14 13:52:29.462509201 +0000 UTC m=+0.019296031 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:52:29.852 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:52:29 smithi008 podman[113026]: 2024-04-14 13:52:29.595085621 +0000 UTC m=+0.151872436 container create d417f5b0255a7c2fbbb4e5977125c54bcdfd00bf0582e06bf4e0fba265ee5d7a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-14T13:52:29.852 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:52:29 smithi008 podman[113026]: 2024-04-14 13:52:29.796156531 +0000 UTC m=+0.352943347 container init d417f5b0255a7c2fbbb4e5977125c54bcdfd00bf0582e06bf4e0fba265ee5d7a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-14T13:52:29.852 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:52:29 smithi008 podman[113026]: 2024-04-14 13:52:29.799508673 +0000 UTC m=+0.356295499 container start d417f5b0255a7c2fbbb4e5977125c54bcdfd00bf0582e06bf4e0fba265ee5d7a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=main-a971fd5) 2024-04-14T13:52:30.299 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:29 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:29.884+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:02.963468+0000 front 2024-04-14T13:52:02.963389+0000 (oldest deadline 2024-04-14T13:52:26.463537+0000) 2024-04-14T13:52:30.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:29 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:29.816+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:30.352 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:52:29 smithi008 podman[113026]: 2024-04-14 13:52:29.888100336 +0000 UTC m=+0.444887161 container attach d417f5b0255a7c2fbbb4e5977125c54bcdfd00bf0582e06bf4e0fba265ee5d7a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5) 2024-04-14T13:52:30.352 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 14 13:52:30 smithi008 podman[113026]: 2024-04-14 13:52:30.096829693 +0000 UTC m=+0.653616509 container died d417f5b0255a7c2fbbb4e5977125c54bcdfd00bf0582e06bf4e0fba265ee5d7a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:52:30.720 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.3.service' 2024-04-14T13:52:30.761 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T13:52:30.761 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-14T13:52:30.761 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-14T13:52:30.761 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.4 2024-04-14T13:52:30.790 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:30 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[58181]: 2024-04-14T13:52:30.357+0000 7f4a427f9700 -1 osd.4 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.685341+0000 front 2024-04-14T13:52:03.685307+0000 (oldest deadline 2024-04-14T13:52:29.585109+0000) 2024-04-14T13:52:30.790 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:30 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:30.587+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:31.049 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:30 smithi012 systemd[1]: Stopping Ceph osd.4 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:52:31.050 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:30 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:30.871+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:02.963468+0000 front 2024-04-14T13:52:02.963389+0000 (oldest deadline 2024-04-14T13:52:26.463537+0000) 2024-04-14T13:52:31.050 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:30 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:30.848+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:31.546 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:31 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[58181]: 2024-04-14T13:52:31.078+0000 7f4a46205700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:52:31.546 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:31 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[58181]: 2024-04-14T13:52:31.078+0000 7f4a46205700 -1 osd.4 154 *** Got signal Terminated *** 2024-04-14T13:52:31.547 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:31 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[58181]: 2024-04-14T13:52:31.078+0000 7f4a46205700 -1 osd.4 154 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T13:52:31.547 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:31 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[58181]: 2024-04-14T13:52:31.323+0000 7f4a427f9700 -1 osd.4 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.685341+0000 front 2024-04-14T13:52:03.685307+0000 (oldest deadline 2024-04-14T13:52:29.585109+0000) 2024-04-14T13:52:31.799 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:31 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:31.544+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:32.299 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:31 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:31.867+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:02.963468+0000 front 2024-04-14T13:52:02.963389+0000 (oldest deadline 2024-04-14T13:52:26.463537+0000) 2024-04-14T13:52:32.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:31 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:31.851+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:32.799 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:32 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[58181]: 2024-04-14T13:52:32.367+0000 7f4a427f9700 -1 osd.4 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.685341+0000 front 2024-04-14T13:52:03.685307+0000 (oldest deadline 2024-04-14T13:52:29.585109+0000) 2024-04-14T13:52:32.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:32 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:32.500+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:33.300 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:32 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:32.846+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:02.963468+0000 front 2024-04-14T13:52:02.963389+0000 (oldest deadline 2024-04-14T13:52:26.463537+0000) 2024-04-14T13:52:33.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:32 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:32.825+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:33.799 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:33 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[58181]: 2024-04-14T13:52:33.412+0000 7f4a427f9700 -1 osd.4 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.685341+0000 front 2024-04-14T13:52:03.685307+0000 (oldest deadline 2024-04-14T13:52:29.585109+0000) 2024-04-14T13:52:33.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:33 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:33.474+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:33.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:33 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:33.474+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:34.299 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:33 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:33.887+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:02.963468+0000 front 2024-04-14T13:52:02.963389+0000 (oldest deadline 2024-04-14T13:52:26.463537+0000) 2024-04-14T13:52:34.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:33 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:33.829+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:34.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:33 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:33.829+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.503389+0000 front 2024-04-14T13:52:09.503336+0000 (oldest deadline 2024-04-14T13:52:33.003194+0000) 2024-04-14T13:52:34.800 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:34 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[58181]: 2024-04-14T13:52:34.403+0000 7f4a427f9700 -1 osd.4 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.685341+0000 front 2024-04-14T13:52:03.685307+0000 (oldest deadline 2024-04-14T13:52:29.585109+0000) 2024-04-14T13:52:34.800 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:34 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[58181]: 2024-04-14T13:52:34.403+0000 7f4a427f9700 -1 osd.4 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.585501+0000 front 2024-04-14T13:52:09.585600+0000 (oldest deadline 2024-04-14T13:52:34.285553+0000) 2024-04-14T13:52:34.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:34 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:34.432+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:34.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:34 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:34.432+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:35.299 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:34 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:34.934+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:02.963468+0000 front 2024-04-14T13:52:02.963389+0000 (oldest deadline 2024-04-14T13:52:26.463537+0000) 2024-04-14T13:52:35.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:34 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:34.872+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:35.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:34 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:34.872+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.503389+0000 front 2024-04-14T13:52:09.503336+0000 (oldest deadline 2024-04-14T13:52:33.003194+0000) 2024-04-14T13:52:35.799 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:35 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:35.394+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:35.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:35 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:35.394+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:35.800 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:35 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[58181]: 2024-04-14T13:52:35.372+0000 7f4a427f9700 -1 osd.4 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.685341+0000 front 2024-04-14T13:52:03.685307+0000 (oldest deadline 2024-04-14T13:52:29.585109+0000) 2024-04-14T13:52:35.800 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:35 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4[58181]: 2024-04-14T13:52:35.372+0000 7f4a427f9700 -1 osd.4 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.585501+0000 front 2024-04-14T13:52:09.585600+0000 (oldest deadline 2024-04-14T13:52:34.285553+0000) 2024-04-14T13:52:36.300 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:35 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:35.917+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:02.963468+0000 front 2024-04-14T13:52:02.963389+0000 (oldest deadline 2024-04-14T13:52:26.463537+0000) 2024-04-14T13:52:36.300 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:35 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:35.917+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:11.665158+0000 front 2024-04-14T13:52:11.665299+0000 (oldest deadline 2024-04-14T13:52:35.765132+0000) 2024-04-14T13:52:36.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:35 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:35.864+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:36.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:35 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:35.864+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.503389+0000 front 2024-04-14T13:52:09.503336+0000 (oldest deadline 2024-04-14T13:52:33.003194+0000) 2024-04-14T13:52:36.740 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:36 smithi012 podman[78322]: 2024-04-14 13:52:36.465751502 +0000 UTC m=+5.514519697 container died 178f9462f5ba00df66f38c91318e0b3f34c609ee840b9e4e72b5414e0ee75d7d (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-14T13:52:36.740 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:36 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:36.374+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:36.740 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:36 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:36.374+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:37.049 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:36 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:36.960+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:02.963468+0000 front 2024-04-14T13:52:02.963389+0000 (oldest deadline 2024-04-14T13:52:26.463537+0000) 2024-04-14T13:52:37.049 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:36 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:36.960+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:11.665158+0000 front 2024-04-14T13:52:11.665299+0000 (oldest deadline 2024-04-14T13:52:35.765132+0000) 2024-04-14T13:52:37.050 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:36 smithi012 podman[78322]: 2024-04-14 13:52:36.936697315 +0000 UTC m=+5.985465499 container cleanup 178f9462f5ba00df66f38c91318e0b3f34c609ee840b9e4e72b5414e0ee75d7d (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-14T13:52:37.050 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:36 smithi012 bash[78322]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4 2024-04-14T13:52:37.050 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:36 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:36.867+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:37.050 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:36 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:36.867+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.503389+0000 front 2024-04-14T13:52:09.503336+0000 (oldest deadline 2024-04-14T13:52:33.003194+0000) 2024-04-14T13:52:37.508 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:37 smithi012 podman[78339]: 2024-04-14 13:52:37.180075095 +0000 UTC m=+0.712608365 container remove 178f9462f5ba00df66f38c91318e0b3f34c609ee840b9e4e72b5414e0ee75d7d (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux ) 2024-04-14T13:52:37.509 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:37 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:37.350+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:37.509 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:37 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:37.350+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:37.768 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:37 smithi012 podman[78428]: 2024-04-14 13:52:37.409920392 +0000 UTC m=+0.018433810 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:52:37.768 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:37 smithi012 podman[78428]: 2024-04-14 13:52:37.514247407 +0000 UTC m=+0.122760866 container create a7634e2af3d24344c586ce8157a7c780b973b5056aeb2eb799613e6e7e91339a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-14T13:52:37.768 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:37 smithi012 podman[78428]: 2024-04-14 13:52:37.699934707 +0000 UTC m=+0.308448124 container init a7634e2af3d24344c586ce8157a7c780b973b5056aeb2eb799613e6e7e91339a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-14T13:52:37.768 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:37 smithi012 podman[78428]: 2024-04-14 13:52:37.702972481 +0000 UTC m=+0.311485898 container start a7634e2af3d24344c586ce8157a7c780b973b5056aeb2eb799613e6e7e91339a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-14T13:52:37.768 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:37 smithi012 podman[78428]: 2024-04-14 13:52:37.766500438 +0000 UTC m=+0.375013856 container attach a7634e2af3d24344c586ce8157a7c780b973b5056aeb2eb799613e6e7e91339a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0) 2024-04-14T13:52:38.050 INFO:journalctl@ceph.osd.4.smithi012.stdout:Apr 14 13:52:38 smithi012 podman[78428]: 2024-04-14 13:52:38.009480853 +0000 UTC m=+0.617994271 container died a7634e2af3d24344c586ce8157a7c780b973b5056aeb2eb799613e6e7e91339a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-14T13:52:38.050 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:38 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:38.008+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:02.963468+0000 front 2024-04-14T13:52:02.963389+0000 (oldest deadline 2024-04-14T13:52:26.463537+0000) 2024-04-14T13:52:38.050 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:38 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:38.008+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:11.665158+0000 front 2024-04-14T13:52:11.665299+0000 (oldest deadline 2024-04-14T13:52:35.765132+0000) 2024-04-14T13:52:38.051 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:37 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:37.843+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:38.051 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:37 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:37.843+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.503389+0000 front 2024-04-14T13:52:09.503336+0000 (oldest deadline 2024-04-14T13:52:33.003194+0000) 2024-04-14T13:52:38.655 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:38 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:38.370+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:38.655 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:38 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:38.370+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:38.677 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.4.service' 2024-04-14T13:52:38.763 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T13:52:38.763 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-14T13:52:38.763 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-14T13:52:38.764 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.5 2024-04-14T13:52:38.992 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:38 smithi012 systemd[1]: Stopping Ceph osd.5 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:52:38.993 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:38 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:38.838+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:38.993 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:38 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:38.838+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.503389+0000 front 2024-04-14T13:52:09.503336+0000 (oldest deadline 2024-04-14T13:52:33.003194+0000) 2024-04-14T13:52:39.299 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:38 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:38.990+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:02.963468+0000 front 2024-04-14T13:52:02.963389+0000 (oldest deadline 2024-04-14T13:52:26.463537+0000) 2024-04-14T13:52:39.299 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:38 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:38.990+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:11.665158+0000 front 2024-04-14T13:52:11.665299+0000 (oldest deadline 2024-04-14T13:52:35.765132+0000) 2024-04-14T13:52:39.299 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:39 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:39.082+0000 7f6e22f3b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:52:39.299 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:39 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:39.082+0000 7f6e22f3b700 -1 osd.5 154 *** Got signal Terminated *** 2024-04-14T13:52:39.300 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:39 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:39.082+0000 7f6e22f3b700 -1 osd.5 154 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T13:52:39.799 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:39 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:39.338+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:39.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:39 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:39.338+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:40.299 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:39 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:39.963+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:02.963468+0000 front 2024-04-14T13:52:02.963389+0000 (oldest deadline 2024-04-14T13:52:26.463537+0000) 2024-04-14T13:52:40.299 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:39 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:39.963+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:11.665158+0000 front 2024-04-14T13:52:11.665299+0000 (oldest deadline 2024-04-14T13:52:35.765132+0000) 2024-04-14T13:52:40.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:39 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:39.845+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:40.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:39 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:39.845+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.503389+0000 front 2024-04-14T13:52:09.503336+0000 (oldest deadline 2024-04-14T13:52:33.003194+0000) 2024-04-14T13:52:40.799 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:40 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:40.323+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:40.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:40 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:40.323+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:41.299 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:40 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:40.921+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:02.963468+0000 front 2024-04-14T13:52:02.963389+0000 (oldest deadline 2024-04-14T13:52:26.463537+0000) 2024-04-14T13:52:41.299 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:40 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:40.921+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:11.665158+0000 front 2024-04-14T13:52:11.665299+0000 (oldest deadline 2024-04-14T13:52:35.765132+0000) 2024-04-14T13:52:41.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:40 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:40.888+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:41.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:40 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:40.888+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.503389+0000 front 2024-04-14T13:52:09.503336+0000 (oldest deadline 2024-04-14T13:52:33.003194+0000) 2024-04-14T13:52:41.799 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:41 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:41.365+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:41.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:41 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:41.365+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:41.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:41 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:41.365+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:52:42.299 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:41 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:41.936+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:42.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:41 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:41.936+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.503389+0000 front 2024-04-14T13:52:09.503336+0000 (oldest deadline 2024-04-14T13:52:33.003194+0000) 2024-04-14T13:52:42.300 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:41 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:41.933+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:02.963468+0000 front 2024-04-14T13:52:02.963389+0000 (oldest deadline 2024-04-14T13:52:26.463537+0000) 2024-04-14T13:52:42.300 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:41 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:41.933+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:11.665158+0000 front 2024-04-14T13:52:11.665299+0000 (oldest deadline 2024-04-14T13:52:35.765132+0000) 2024-04-14T13:52:42.300 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:41 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:41.933+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:17.465963+0000 front 2024-04-14T13:52:17.465988+0000 (oldest deadline 2024-04-14T13:52:40.965940+0000) 2024-04-14T13:52:42.799 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:42 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:42.349+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:42.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:42 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:42.349+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:42.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:42 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:42.349+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:52:43.299 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:42 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:42.956+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:43.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:42 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:42.956+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.503389+0000 front 2024-04-14T13:52:09.503336+0000 (oldest deadline 2024-04-14T13:52:33.003194+0000) 2024-04-14T13:52:43.300 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:42 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:42.888+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:02.963468+0000 front 2024-04-14T13:52:02.963389+0000 (oldest deadline 2024-04-14T13:52:26.463537+0000) 2024-04-14T13:52:43.300 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:42 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:42.888+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:11.665158+0000 front 2024-04-14T13:52:11.665299+0000 (oldest deadline 2024-04-14T13:52:35.765132+0000) 2024-04-14T13:52:43.300 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:42 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:42.888+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:17.465963+0000 front 2024-04-14T13:52:17.465988+0000 (oldest deadline 2024-04-14T13:52:40.965940+0000) 2024-04-14T13:52:43.799 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:43 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:43.391+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:43.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:43 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:43.391+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:43.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:43 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:43.391+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:52:44.300 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:43 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:43.861+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:02.963468+0000 front 2024-04-14T13:52:02.963389+0000 (oldest deadline 2024-04-14T13:52:26.463537+0000) 2024-04-14T13:52:44.300 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:43 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:43.861+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:11.665158+0000 front 2024-04-14T13:52:11.665299+0000 (oldest deadline 2024-04-14T13:52:35.765132+0000) 2024-04-14T13:52:44.300 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:43 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5[60436]: 2024-04-14T13:52:43.861+0000 7f6e1b527700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:17.465963+0000 front 2024-04-14T13:52:17.465988+0000 (oldest deadline 2024-04-14T13:52:40.965940+0000) 2024-04-14T13:52:44.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:43 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:43.974+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:44.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:43 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:43.974+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.503389+0000 front 2024-04-14T13:52:09.503336+0000 (oldest deadline 2024-04-14T13:52:33.003194+0000) 2024-04-14T13:52:44.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:43 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:43.974+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.204749+0000 front 2024-04-14T13:52:18.204843+0000 (oldest deadline 2024-04-14T13:52:43.504684+0000) 2024-04-14T13:52:44.799 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:44 smithi012 podman[78535]: 2024-04-14 13:52:44.638266191 +0000 UTC m=+5.691457077 container died 25e416621d4036239dc419587544f348c677403acfb703c339ba78306ceb4e3e (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-14T13:52:44.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:44 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:44.421+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:44.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:44 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:44.421+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:44.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:44 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:44.421+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:52:45.285 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:45 smithi012 podman[78535]: 2024-04-14 13:52:45.057120314 +0000 UTC m=+6.110311243 container cleanup 25e416621d4036239dc419587544f348c677403acfb703c339ba78306ceb4e3e (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:52:45.285 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:45 smithi012 bash[78535]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5 2024-04-14T13:52:45.285 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:45 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:45.015+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:45.285 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:45 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:45.015+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.503389+0000 front 2024-04-14T13:52:09.503336+0000 (oldest deadline 2024-04-14T13:52:33.003194+0000) 2024-04-14T13:52:45.286 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:45 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:45.015+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.204749+0000 front 2024-04-14T13:52:18.204843+0000 (oldest deadline 2024-04-14T13:52:43.504684+0000) 2024-04-14T13:52:45.549 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:45 smithi012 podman[78586]: 2024-04-14 13:52:45.283674425 +0000 UTC m=+0.641840335 container remove 25e416621d4036239dc419587544f348c677403acfb703c339ba78306ceb4e3e (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T13:52:45.549 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:45 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:45.388+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:45.550 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:45 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:45.388+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:45.550 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:45 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:45.388+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:52:46.026 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:45 smithi012 podman[78679]: 2024-04-14 13:52:45.726304283 +0000 UTC m=+0.112979526 container create 333814d418bc3f7c4691b390069cc9053ce057e8c5cdb8abc3fe99b9dae8e6d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD) 2024-04-14T13:52:46.026 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:45 smithi012 podman[78679]: 2024-04-14 13:52:45.631222282 +0000 UTC m=+0.017897483 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:52:46.026 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:45 smithi012 podman[78679]: 2024-04-14 13:52:45.945843317 +0000 UTC m=+0.332518530 container init 333814d418bc3f7c4691b390069cc9053ce057e8c5cdb8abc3fe99b9dae8e6d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-14T13:52:46.026 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:45 smithi012 podman[78679]: 2024-04-14 13:52:45.949148934 +0000 UTC m=+0.335824144 container start 333814d418bc3f7c4691b390069cc9053ce057e8c5cdb8abc3fe99b9dae8e6d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-14T13:52:46.300 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:46 smithi012 podman[78679]: 2024-04-14 13:52:46.028473166 +0000 UTC m=+0.415148377 container attach 333814d418bc3f7c4691b390069cc9053ce057e8c5cdb8abc3fe99b9dae8e6d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-14T13:52:46.300 INFO:journalctl@ceph.osd.5.smithi012.stdout:Apr 14 13:52:46 smithi012 podman[78679]: 2024-04-14 13:52:46.246346775 +0000 UTC m=+0.633021976 container died 333814d418bc3f7c4691b390069cc9053ce057e8c5cdb8abc3fe99b9dae8e6d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-14T13:52:46.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:46 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:46.024+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:46.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:46 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:46.024+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.503389+0000 front 2024-04-14T13:52:09.503336+0000 (oldest deadline 2024-04-14T13:52:33.003194+0000) 2024-04-14T13:52:46.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:46 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:46.024+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.204749+0000 front 2024-04-14T13:52:18.204843+0000 (oldest deadline 2024-04-14T13:52:43.504684+0000) 2024-04-14T13:52:46.783 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:46 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:46.351+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:46.783 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:46 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:46.351+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:46.783 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:46 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:46.351+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:52:46.808 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.5.service' 2024-04-14T13:52:46.849 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T13:52:46.849 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-14T13:52:46.849 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-14T13:52:46.849 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.6 2024-04-14T13:52:47.046 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:46 smithi012 systemd[1]: Stopping Ceph osd.6 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:52:47.299 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:47 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:47.043+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:47.299 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:47 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:47.043+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.503389+0000 front 2024-04-14T13:52:09.503336+0000 (oldest deadline 2024-04-14T13:52:33.003194+0000) 2024-04-14T13:52:47.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:47 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:47.043+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.204749+0000 front 2024-04-14T13:52:18.204843+0000 (oldest deadline 2024-04-14T13:52:43.504684+0000) 2024-04-14T13:52:47.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:47 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:47.160+0000 7f3b13fca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:52:47.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:47 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:47.160+0000 7f3b13fca700 -1 osd.6 154 *** Got signal Terminated *** 2024-04-14T13:52:47.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:47 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:47.160+0000 7f3b13fca700 -1 osd.6 154 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T13:52:47.799 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:47 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:47.390+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:47.799 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:47 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:47.390+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:47.799 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:47 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:47.390+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:52:48.437 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:48 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:48.073+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:48.437 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:48 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:48.073+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.503389+0000 front 2024-04-14T13:52:09.503336+0000 (oldest deadline 2024-04-14T13:52:33.003194+0000) 2024-04-14T13:52:48.437 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:48 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:48.073+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.204749+0000 front 2024-04-14T13:52:18.204843+0000 (oldest deadline 2024-04-14T13:52:43.504684+0000) 2024-04-14T13:52:48.799 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:48 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:48.435+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:48.799 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:48 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:48.435+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:48.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:48 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:48.435+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:52:49.412 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:49 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:49.067+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:49.412 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:49 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:49.067+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.503389+0000 front 2024-04-14T13:52:09.503336+0000 (oldest deadline 2024-04-14T13:52:33.003194+0000) 2024-04-14T13:52:49.412 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:49 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:49.067+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.204749+0000 front 2024-04-14T13:52:18.204843+0000 (oldest deadline 2024-04-14T13:52:43.504684+0000) 2024-04-14T13:52:49.412 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:49 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:49.067+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-14T13:52:25.805655+0000 front 2024-04-14T13:52:25.805682+0000 (oldest deadline 2024-04-14T13:52:48.705500+0000) 2024-04-14T13:52:49.799 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:49 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:49.410+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:49.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:49 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:49.410+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:49.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:49 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:49.410+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:52:49.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:49 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:49.410+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-14T13:52:26.903764+0000 front 2024-04-14T13:52:26.903779+0000 (oldest deadline 2024-04-14T13:52:49.203555+0000) 2024-04-14T13:52:50.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:50 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:50.025+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:50.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:50 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:50.025+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.503389+0000 front 2024-04-14T13:52:09.503336+0000 (oldest deadline 2024-04-14T13:52:33.003194+0000) 2024-04-14T13:52:50.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:50 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:50.025+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.204749+0000 front 2024-04-14T13:52:18.204843+0000 (oldest deadline 2024-04-14T13:52:43.504684+0000) 2024-04-14T13:52:50.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:50 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:50.025+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-14T13:52:25.805655+0000 front 2024-04-14T13:52:25.805682+0000 (oldest deadline 2024-04-14T13:52:48.705500+0000) 2024-04-14T13:52:50.799 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:50 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:50.381+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:50.799 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:50 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:50.381+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:50.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:50 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:50.381+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:52:50.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:50 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:50.381+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-14T13:52:26.903764+0000 front 2024-04-14T13:52:26.903779+0000 (oldest deadline 2024-04-14T13:52:49.203555+0000) 2024-04-14T13:52:51.299 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:50 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:50.989+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:51.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:50 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:50.989+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.503389+0000 front 2024-04-14T13:52:09.503336+0000 (oldest deadline 2024-04-14T13:52:33.003194+0000) 2024-04-14T13:52:51.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:50 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:50.989+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.204749+0000 front 2024-04-14T13:52:18.204843+0000 (oldest deadline 2024-04-14T13:52:43.504684+0000) 2024-04-14T13:52:51.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:50 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:50.989+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-14T13:52:25.805655+0000 front 2024-04-14T13:52:25.805682+0000 (oldest deadline 2024-04-14T13:52:48.705500+0000) 2024-04-14T13:52:51.799 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:51 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:51.392+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:51.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:51 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:51.392+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:51.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:51 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:51.392+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:52:51.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:51 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:51.392+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-14T13:52:26.903764+0000 front 2024-04-14T13:52:26.903779+0000 (oldest deadline 2024-04-14T13:52:49.203555+0000) 2024-04-14T13:52:52.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:52 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:52.004+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:03.801810+0000 front 2024-04-14T13:52:03.801828+0000 (oldest deadline 2024-04-14T13:52:26.101879+0000) 2024-04-14T13:52:52.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:52 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:52.004+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.503389+0000 front 2024-04-14T13:52:09.503336+0000 (oldest deadline 2024-04-14T13:52:33.003194+0000) 2024-04-14T13:52:52.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:52 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:52.004+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.204749+0000 front 2024-04-14T13:52:18.204843+0000 (oldest deadline 2024-04-14T13:52:43.504684+0000) 2024-04-14T13:52:52.300 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:52 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6[62697]: 2024-04-14T13:52:52.004+0000 7f3b105be700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-14T13:52:25.805655+0000 front 2024-04-14T13:52:25.805682+0000 (oldest deadline 2024-04-14T13:52:48.705500+0000) 2024-04-14T13:52:52.754 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:52 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:52.345+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:52.754 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:52 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:52.345+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:52.754 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:52 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:52.345+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:52:52.754 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:52 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:52.345+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-14T13:52:26.903764+0000 front 2024-04-14T13:52:26.903779+0000 (oldest deadline 2024-04-14T13:52:49.203555+0000) 2024-04-14T13:52:53.045 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:52 smithi012 podman[78784]: 2024-04-14 13:52:52.75250162 +0000 UTC m=+5.683105835 container died 97f0b691cf873bce2a3869deabb10117219eb6b300a1d0bc999b4b8a128ea11a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403) 2024-04-14T13:52:53.299 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:53 smithi012 podman[78784]: 2024-04-14 13:52:53.259369958 +0000 UTC m=+6.189974172 container cleanup 97f0b691cf873bce2a3869deabb10117219eb6b300a1d0bc999b4b8a128ea11a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T13:52:53.299 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:53 smithi012 bash[78784]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6 2024-04-14T13:52:53.799 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:53 smithi012 podman[78801]: 2024-04-14 13:52:53.502188577 +0000 UTC m=+0.744458883 container remove 97f0b691cf873bce2a3869deabb10117219eb6b300a1d0bc999b4b8a128ea11a (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-14T13:52:53.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:53 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:53.393+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:53.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:53 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:53.393+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:53.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:53 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:53.393+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:52:53.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:53 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:53.393+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-14T13:52:26.903764+0000 front 2024-04-14T13:52:26.903779+0000 (oldest deadline 2024-04-14T13:52:49.203555+0000) 2024-04-14T13:52:54.199 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:53 smithi012 podman[78890]: 2024-04-14 13:52:53.93652156 +0000 UTC m=+0.113534194 container create 7c57af562dad6ef15aa5502634181691afe8a592c449a7a9232db01788872124 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_CLEAN=True) 2024-04-14T13:52:54.199 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:53 smithi012 podman[78890]: 2024-04-14 13:52:53.840978891 +0000 UTC m=+0.017991503 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:52:54.199 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:54 smithi012 podman[78890]: 2024-04-14 13:52:54.197905039 +0000 UTC m=+0.374917638 container init 7c57af562dad6ef15aa5502634181691afe8a592c449a7a9232db01788872124 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T13:52:54.497 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:54 smithi012 podman[78890]: 2024-04-14 13:52:54.201480336 +0000 UTC m=+0.378492929 container start 7c57af562dad6ef15aa5502634181691afe8a592c449a7a9232db01788872124 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T13:52:54.497 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:54 smithi012 podman[78890]: 2024-04-14 13:52:54.297528011 +0000 UTC m=+0.474540606 container attach 7c57af562dad6ef15aa5502634181691afe8a592c449a7a9232db01788872124 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-14T13:52:54.497 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:54 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:54.379+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:54.497 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:54 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:54.379+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:54.497 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:54 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:54.379+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:52:54.497 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:54 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:54.379+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-14T13:52:26.903764+0000 front 2024-04-14T13:52:26.903779+0000 (oldest deadline 2024-04-14T13:52:49.203555+0000) 2024-04-14T13:52:54.793 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 14 13:52:54 smithi012 podman[78890]: 2024-04-14 13:52:54.495704686 +0000 UTC m=+0.672717283 container died 7c57af562dad6ef15aa5502634181691afe8a592c449a7a9232db01788872124 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-14T13:52:55.065 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.6.service' 2024-04-14T13:52:55.108 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T13:52:55.109 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-14T13:52:55.109 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-14T13:52:55.109 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.7 2024-04-14T13:52:55.299 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:55 smithi012 systemd[1]: Stopping Ceph osd.7 for a8b03f6a-fa62-11ee-bc8e-c7b262605968... 2024-04-14T13:52:55.799 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:55 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:55.348+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:55.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:55 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:55.348+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:55.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:55 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:55.348+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:52:55.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:55 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:55.348+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-14T13:52:26.903764+0000 front 2024-04-14T13:52:26.903779+0000 (oldest deadline 2024-04-14T13:52:49.203555+0000) 2024-04-14T13:52:55.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:55 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:55.478+0000 7faec464d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:52:55.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:55 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:55.478+0000 7faec464d700 -1 osd.7 154 *** Got signal Terminated *** 2024-04-14T13:52:55.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:55 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:55.478+0000 7faec464d700 -1 osd.7 154 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T13:52:56.799 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:56 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:56.365+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:56.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:56 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:56.365+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:56.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:56 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:56.365+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:52:56.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:56 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:56.365+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-14T13:52:26.903764+0000 front 2024-04-14T13:52:26.903779+0000 (oldest deadline 2024-04-14T13:52:49.203555+0000) 2024-04-14T13:52:57.701 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:57 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:57.395+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:57.701 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:57 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:57.395+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:57.701 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:57 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:57.395+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:52:57.701 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:57 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:57.395+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-14T13:52:26.903764+0000 front 2024-04-14T13:52:26.903779+0000 (oldest deadline 2024-04-14T13:52:49.203555+0000) 2024-04-14T13:52:57.701 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:57 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:57.395+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.12:6804 osd.4 since back 2024-04-14T13:52:32.104334+0000 front 2024-04-14T13:52:32.104407+0000 (oldest deadline 2024-04-14T13:52:56.804337+0000) 2024-04-14T13:52:58.716 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:58 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:58.347+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:58.716 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:58 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:58.347+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:58.717 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:58 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:58.347+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:52:58.717 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:58 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:58.347+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-14T13:52:26.903764+0000 front 2024-04-14T13:52:26.903779+0000 (oldest deadline 2024-04-14T13:52:49.203555+0000) 2024-04-14T13:52:58.717 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:58 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:58.347+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.12:6804 osd.4 since back 2024-04-14T13:52:32.104334+0000 front 2024-04-14T13:52:32.104407+0000 (oldest deadline 2024-04-14T13:52:56.804337+0000) 2024-04-14T13:52:59.549 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:59 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:59.320+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:52:59.550 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:59 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:59.320+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:52:59.550 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:59 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:59.320+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:52:59.550 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:59 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:59.320+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-14T13:52:26.903764+0000 front 2024-04-14T13:52:26.903779+0000 (oldest deadline 2024-04-14T13:52:49.203555+0000) 2024-04-14T13:52:59.550 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:52:59 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:52:59.320+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.12:6804 osd.4 since back 2024-04-14T13:52:32.104334+0000 front 2024-04-14T13:52:32.104407+0000 (oldest deadline 2024-04-14T13:52:56.804337+0000) 2024-04-14T13:53:00.549 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:53:00 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:53:00.285+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-14T13:52:01.900007+0000 front 2024-04-14T13:52:01.899897+0000 (oldest deadline 2024-04-14T13:52:25.399987+0000) 2024-04-14T13:53:00.550 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:53:00 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:53:00.285+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-14T13:52:09.501013+0000 front 2024-04-14T13:52:09.501026+0000 (oldest deadline 2024-04-14T13:52:32.901483+0000) 2024-04-14T13:53:00.550 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:53:00 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:53:00.285+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-14T13:52:18.802395+0000 front 2024-04-14T13:52:18.802425+0000 (oldest deadline 2024-04-14T13:52:41.102315+0000) 2024-04-14T13:53:00.550 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:53:00 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:53:00.285+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-14T13:52:26.903764+0000 front 2024-04-14T13:52:26.903779+0000 (oldest deadline 2024-04-14T13:52:49.203555+0000) 2024-04-14T13:53:00.550 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:53:00 smithi012 ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7[64916]: 2024-04-14T13:53:00.285+0000 7faebcc39700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.12:6804 osd.4 since back 2024-04-14T13:52:32.104334+0000 front 2024-04-14T13:52:32.104407+0000 (oldest deadline 2024-04-14T13:52:56.804337+0000) 2024-04-14T13:53:01.272 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:53:01 smithi012 podman[78994]: 2024-04-14 13:53:01.015135531 +0000 UTC m=+5.680052816 container died 7e4e731dec316eda365169f1085b788085bedd2dbd95fb07eee3be49bc11843c (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux ) 2024-04-14T13:53:01.549 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:53:01 smithi012 podman[78994]: 2024-04-14 13:53:01.369658964 +0000 UTC m=+6.034576247 container cleanup 7e4e731dec316eda365169f1085b788085bedd2dbd95fb07eee3be49bc11843c (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-14T13:53:01.549 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:53:01 smithi012 bash[78994]: ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7 2024-04-14T13:53:01.992 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:53:01 smithi012 podman[79025]: 2024-04-14 13:53:01.645834972 +0000 UTC m=+0.627127491 container remove 7e4e731dec316eda365169f1085b788085bedd2dbd95fb07eee3be49bc11843c (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD) 2024-04-14T13:53:02.299 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:53:01 smithi012 podman[79114]: 2024-04-14 13:53:01.893965224 +0000 UTC m=+0.017826010 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608 2024-04-14T13:53:02.299 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:53:01 smithi012 podman[79114]: 2024-04-14 13:53:01.996791571 +0000 UTC m=+0.120652393 container create d146d04a6860cdd07d505e88792655a7c76779be5c1eef0c3446907d0b4f7159 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux ) 2024-04-14T13:53:02.300 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:53:02 smithi012 podman[79114]: 2024-04-14 13:53:02.232620853 +0000 UTC m=+0.356481640 container init d146d04a6860cdd07d505e88792655a7c76779be5c1eef0c3446907d0b4f7159 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=) 2024-04-14T13:53:02.300 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:53:02 smithi012 podman[79114]: 2024-04-14 13:53:02.235752223 +0000 UTC m=+0.359613005 container start d146d04a6860cdd07d505e88792655a7c76779be5c1eef0c3446907d0b4f7159 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403) 2024-04-14T13:53:02.799 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:53:02 smithi012 podman[79114]: 2024-04-14 13:53:02.315840229 +0000 UTC m=+0.439701011 container attach d146d04a6860cdd07d505e88792655a7c76779be5c1eef0c3446907d0b4f7159 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403) 2024-04-14T13:53:02.800 INFO:journalctl@ceph.osd.7.smithi012.stdout:Apr 14 13:53:02 smithi012 podman[79114]: 2024-04-14 13:53:02.538858066 +0000 UTC m=+0.662718843 container died d146d04a6860cdd07d505e88792655a7c76779be5c1eef0c3446907d0b4f7159 (image=quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608, name=ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-04-14T13:53:03.146 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a8b03f6a-fa62-11ee-bc8e-c7b262605968@osd.7.service' 2024-04-14T13:53:03.188 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T13:53:03.189 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-14T13:53:03.189 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 --force --keep-logs 2024-04-14T13:56:02.708 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 --force --keep-logs 2024-04-14T13:56:24.148 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-14T13:56:24.186 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-14T13:56:24.218 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-14T13:56:24.220 DEBUG:teuthology.misc:Transferring archived files from smithi008:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7645991/remote/smithi008/crash 2024-04-14T13:56:24.221 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar c -f - -C /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/crash -- . 2024-04-14T13:56:24.270 INFO:teuthology.orchestra.run.smithi008.stderr:tar: /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/crash: Cannot open: No such file or directory 2024-04-14T13:56:24.270 INFO:teuthology.orchestra.run.smithi008.stderr:tar: Error is not recoverable: exiting now 2024-04-14T13:56:24.271 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7645991/remote/smithi012/crash 2024-04-14T13:56:24.272 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/crash -- . 2024-04-14T13:56:24.310 INFO:teuthology.orchestra.run.smithi012.stderr:tar: /var/lib/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/crash: Cannot open: No such file or directory 2024-04-14T13:56:24.310 INFO:teuthology.orchestra.run.smithi012.stderr:tar: Error is not recoverable: exiting now 2024-04-14T13:56:24.311 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-14T13:56:24.311 DEBUG:teuthology.orchestra.run.smithi008:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-14T13:56:24.342 INFO:teuthology.orchestra.run.smithi008.stdout:2024-04-14T13:39:35.441183+0000 mon.smithi008 (mon.0) 863 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T13:56:24.343 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-14T13:56:24.343 DEBUG:teuthology.orchestra.run.smithi008:> sudo egrep '\[SEC\]' /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-14T13:56:24.414 DEBUG:teuthology.orchestra.run.smithi008:> sudo egrep '\[ERR\]' /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-14T13:56:24.485 DEBUG:teuthology.orchestra.run.smithi008:> sudo egrep '\[WRN\]' /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-14T13:56:24.553 INFO:teuthology.orchestra.run.smithi008.stdout:2024-04-14T13:39:35.441183+0000 mon.smithi008 (mon.0) 863 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T13:56:24.554 INFO:tasks.cephadm:Compressing logs... 2024-04-14T13:56:24.555 DEBUG:teuthology.orchestra.run.smithi008:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-14T13:56:24.598 DEBUG:teuthology.orchestra.run.smithi012:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-14T13:56:24.625 INFO:teuthology.orchestra.run.smithi008.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-14T13:56:24.625 INFO:teuthology.orchestra.run.smithi008.stderr:: No such file or directory 2024-04-14T13:56:24.625 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-mds.foofs.smithi008.wqbvfi.log 2024-04-14T13:56:24.625 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-mgr.smithi008.yemlhr.log 2024-04-14T13:56:24.625 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-mds.foofs.smithi008.wqbvfi.log: gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.audit.log 2024-04-14T13:56:24.625 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-mgr.smithi008.yemlhr.log: gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-mon.smithi008.log 2024-04-14T13:56:24.625 INFO:teuthology.orchestra.run.smithi008.stderr: 89.2% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-mds.foofs.smithi008.wqbvfi.log.gz 2024-04-14T13:56:24.626 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.1.log 2024-04-14T13:56:24.626 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.audit.log: /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-mon.smithi008.log: gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.3.log 2024-04-14T13:56:24.626 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.0.log 2024-04-14T13:56:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-14T13:56:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:: No such file or directory 2024-04-14T13:56:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.6.log 2024-04-14T13:56:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.audit.log 2024-04-14T13:56:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.4.log 2024-04-14T13:56:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.7.log 2024-04-14T13:56:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.5.log 2024-04-14T13:56:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.4.log: /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-mds.foofs.smithi012.vjovzp.log 2024-04-14T13:56:24.630 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.cephadm.log 2024-04-14T13:56:24.630 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.log 2024-04-14T13:56:24.630 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.cephadm.log: 85.8% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.cephadm.log.gz 2024-04-14T13:56:24.630 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.2.log 2024-04-14T13:56:24.631 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.log: 91.5% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.audit.log.gz 2024-04-14T13:56:24.631 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-volume.log 2024-04-14T13:56:24.633 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-mon.smithi012.log 2024-04-14T13:56:24.633 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-mds.foofs.smithi012.vjovzp.log: /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-mon.smithi012.log: 87.6% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-mds.foofs.smithi012.vjovzp.log.gz 2024-04-14T13:56:24.633 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.cephadm.log 2024-04-14T13:56:24.634 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-mgr.smithi012.ksvnxm.log 2024-04-14T13:56:24.634 INFO:teuthology.orchestra.run.smithi012.stderr: 91.7% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.audit.log.gz 2024-04-14T13:56:24.634 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.cephadm.log: 85.8% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.cephadm.log.gz 2024-04-14T13:56:24.634 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.log 2024-04-14T13:56:24.634 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-mgr.smithi012.ksvnxm.log: gzip -5 --verbose -- /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-volume.log 2024-04-14T13:56:24.634 INFO:teuthology.orchestra.run.smithi012.stderr: 90.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-14T13:56:24.634 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.2.log: 87.2% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.log.gz 2024-04-14T13:56:24.636 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.log: 87.2% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph.log.gz 2024-04-14T13:56:24.651 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-volume.log: 91.4% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-volume.log.gz 2024-04-14T13:56:24.653 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-volume.log: 91.4% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-volume.log.gz 2024-04-14T13:56:24.664 INFO:teuthology.orchestra.run.smithi008.stderr: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-14T13:56:24.706 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-mgr.smithi012.ksvnxm.log.gz 2024-04-14T13:56:24.911 INFO:teuthology.orchestra.run.smithi008.stderr: 88.4% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-mgr.smithi008.yemlhr.log.gz 2024-04-14T13:56:24.958 INFO:teuthology.orchestra.run.smithi012.stderr: 92.6% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-mon.smithi012.log.gz 2024-04-14T13:56:26.096 INFO:teuthology.orchestra.run.smithi008.stderr: 91.9% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-mon.smithi008.log.gz 2024-04-14T13:56:36.541 INFO:teuthology.orchestra.run.smithi008.stderr: 94.5% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.2.log.gz 2024-04-14T13:56:41.108 INFO:teuthology.orchestra.run.smithi012.stderr: 93.9% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.4.log.gz 2024-04-14T13:56:42.359 INFO:teuthology.orchestra.run.smithi012.stderr: 94.3% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.6.log.gz 2024-04-14T13:56:42.389 INFO:teuthology.orchestra.run.smithi008.stderr: 94.1% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.0.log.gz 2024-04-14T13:56:44.110 INFO:teuthology.orchestra.run.smithi008.stderr: 94.2% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.1.log.gz 2024-04-14T13:56:45.328 INFO:teuthology.orchestra.run.smithi012.stderr: 94.2% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.5.log.gz 2024-04-14T13:56:46.311 INFO:teuthology.orchestra.run.smithi008.stderr: 94.1% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.3.log.gz 2024-04-14T13:56:46.312 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-14T13:56:46.313 INFO:teuthology.orchestra.run.smithi008.stderr:real 0m21.702s 2024-04-14T13:56:46.313 INFO:teuthology.orchestra.run.smithi008.stderr:user 1m10.072s 2024-04-14T13:56:46.313 INFO:teuthology.orchestra.run.smithi008.stderr:sys 0m2.504s 2024-04-14T13:56:51.227 INFO:teuthology.orchestra.run.smithi012.stderr: 94.1% -- replaced with /var/log/ceph/a8b03f6a-fa62-11ee-bc8e-c7b262605968/ceph-osd.7.log.gz 2024-04-14T13:56:51.227 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-04-14T13:56:51.227 INFO:teuthology.orchestra.run.smithi012.stderr:real 0m26.613s 2024-04-14T13:56:51.227 INFO:teuthology.orchestra.run.smithi012.stderr:user 1m18.877s 2024-04-14T13:56:51.227 INFO:teuthology.orchestra.run.smithi012.stderr:sys 0m2.942s 2024-04-14T13:56:51.227 INFO:tasks.cephadm:Archiving logs... 2024-04-14T13:56:51.228 DEBUG:teuthology.misc:Transferring archived files from smithi008:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7645991/remote/smithi008/log 2024-04-14T13:56:51.229 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-14T13:57:00.733 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7645991/remote/smithi012/log 2024-04-14T13:57:00.734 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-14T13:57:11.862 INFO:tasks.cephadm:Removing cluster... 2024-04-14T13:57:11.862 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 --force 2024-04-14T13:57:12.243 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a8b03f6a-fa62-11ee-bc8e-c7b262605968 --force 2024-04-14T13:57:12.637 INFO:tasks.cephadm:Removing cephadm ... 2024-04-14T13:57:12.637 DEBUG:teuthology.orchestra.run.smithi008:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-14T13:57:12.652 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-14T13:57:12.669 INFO:tasks.cephadm:Teardown complete 2024-04-14T13:57:12.669 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-14T13:57:12.682 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-14T13:57:12.682 DEBUG:teuthology.orchestra.run.smithi008:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T13:57:12.694 DEBUG:teuthology.orchestra.run.smithi012:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T13:57:12.705 INFO:teuthology.orchestra.run.smithi008.stderr:bash: line 1: ntpq: command not found 2024-04-14T13:57:12.710 INFO:teuthology.orchestra.run.smithi008.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T13:57:12.710 INFO:teuthology.orchestra.run.smithi008.stdout:=============================================================================== 2024-04-14T13:57:12.710 INFO:teuthology.orchestra.run.smithi008.stdout:^* hv01.front.sepia.ceph.com 3 6 377 1 +774us[ +805us] +/- 65ms 2024-04-14T13:57:12.710 INFO:teuthology.orchestra.run.smithi008.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 2 -458us[ -427us] +/- 55ms 2024-04-14T13:57:12.710 INFO:teuthology.orchestra.run.smithi008.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -210us[ -187us] +/- 66ms 2024-04-14T13:57:12.710 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T13:57:12.720 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2024-04-14T13:57:12.723 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T13:57:12.723 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-04-14T13:57:12.724 INFO:teuthology.orchestra.run.smithi012.stdout:^* hv01.front.sepia.ceph.com 3 6 377 0 +757us[ +776us] +/- 65ms 2024-04-14T13:57:12.724 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 65 -460us[ -446us] +/- 54ms 2024-04-14T13:57:12.724 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 -189us[ -170us] +/- 67ms 2024-04-14T13:57:12.724 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T13:57:12.725 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-14T13:57:12.735 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-14T13:57:12.736 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-14T13:57:12.745 DEBUG:teuthology.orchestra.run.smithi008:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-14T13:57:12.835 DEBUG:teuthology.orchestra.run.smithi012:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-14T13:57:12.914 DEBUG:teuthology.orchestra.run.smithi008:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-14T13:57:12.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T13:57:12.944 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-14T13:57:12.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T13:57:12.975 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-14T13:57:12.986 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-14T13:57:12.996 INFO:teuthology.task.internal:Duration was 2091.898464 seconds 2024-04-14T13:57:12.997 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-14T13:57:13.088 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-14T13:57:13.088 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-14T13:57:13.091 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-14T13:57:13.132 INFO:teuthology.orchestra.run.smithi008.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T13:57:13.133 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T13:57:13.637 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-14T13:57:13.637 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi008.front.sepia.ceph.com 2024-04-14T13:57:13.637 DEBUG:teuthology.orchestra.run.smithi008:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-14T13:57:13.698 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi012.front.sepia.ceph.com 2024-04-14T13:57:13.699 DEBUG:teuthology.orchestra.run.smithi012:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-14T13:57:13.725 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-14T13:57:13.725 DEBUG:teuthology.orchestra.run.smithi008:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-14T13:57:13.741 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-14T13:57:13.794 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-14T13:57:13.794 DEBUG:teuthology.orchestra.run.smithi008:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-14T13:57:13.810 DEBUG:teuthology.orchestra.run.smithi012:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-14T13:57:27.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-14T13:57:27.796 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-14T13:57:27.796 DEBUG:teuthology.orchestra.run.smithi008:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-14T13:57:27.823 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-14T13:57:27.855 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-14T13:57:27.867 DEBUG:teuthology.orchestra.run.smithi008:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-14T13:57:27.869 DEBUG:teuthology.orchestra.run.smithi012:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-14T13:57:27.892 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern = core 2024-04-14T13:57:27.927 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = core 2024-04-14T13:57:27.957 DEBUG:teuthology.orchestra.run.smithi008:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-14T13:57:27.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T13:57:27.986 DEBUG:teuthology.orchestra.run.smithi012:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-14T13:57:28.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T13:57:28.013 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-14T13:57:28.025 INFO:teuthology.task.internal:Transferring archived files... 2024-04-14T13:57:28.026 DEBUG:teuthology.misc:Transferring archived files from smithi008:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7645991/remote/smithi008 2024-04-14T13:57:28.026 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-14T13:57:28.092 DEBUG:teuthology.misc:Transferring archived files from smithi012:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7645991/remote/smithi012 2024-04-14T13:57:28.092 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-14T13:57:28.137 INFO:teuthology.task.internal:Removing archive directory... 2024-04-14T13:57:28.137 DEBUG:teuthology.orchestra.run.smithi008:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-14T13:57:28.140 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-14T13:57:28.186 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-14T13:57:28.201 INFO:teuthology.task.internal:Not uploading archives. 2024-04-14T13:57:28.202 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-14T13:57:28.214 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-14T13:57:28.214 DEBUG:teuthology.orchestra.run.smithi008:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-14T13:57:28.219 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-14T13:57:28.231 INFO:teuthology.orchestra.run.smithi008.stdout: 265306 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 14 13:57 /home/ubuntu/cephtest 2024-04-14T13:57:28.243 INFO:teuthology.orchestra.run.smithi012.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 14 13:57 /home/ubuntu/cephtest 2024-04-14T13:57:28.243 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-14T13:57:28.255 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-14T13:57:28.415 INFO:teuthology.nuke:Checking targets against current locks 2024-04-14T13:57:28.434 DEBUG:teuthology.nuke:shortname: smithi008 2024-04-14T13:57:28.435 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-14T13:57:28.456 DEBUG:teuthology.nuke:shortname: smithi012 2024-04-14T13:57:28.456 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-14T13:57:28.459 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi008.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7645991', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 13:14:04.478500', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIXjAFp/v4W4GpwJgDxyfDEBzUFsBxOr07eZaob/n4rP5YVh6fLBGfDMAniPdFOdWaf58tdvIwtvmcQ7l4UmNTA='} 2024-04-14T13:57:28.484 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7645991', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 13:14:04.479663', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIF6RKBD8smBvL6BjGpnEPPv3WDege88/LB1Swg7VaJ7JpEvMVNVwmrhlV9DQ7UXvaPY4PmD6/47087f32bS5+4='} 2024-04-14T13:57:28.516 INFO:teuthology.orchestra.console.smithi008:Power off 2024-04-14T13:57:28.516 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-14T13:57:28.533 INFO:teuthology.orchestra.console.smithi012:Power off 2024-04-14T13:57:28.534 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-14T13:57:28.543 DEBUG:teuthology.orchestra.console.smithi008:power off output: Chassis Power Control: Down/Off 2024-04-14T13:57:28.543 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T13:57:28.559 DEBUG:teuthology.orchestra.console.smithi012:power off output: Chassis Power Control: Down/Off 2024-04-14T13:57:28.559 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T13:57:28.569 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is on 2024-04-14T13:57:28.583 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-04-14T13:57:32.570 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T13:57:32.584 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T13:57:32.681 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is on 2024-04-14T13:57:32.693 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-04-14T13:57:36.686 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T13:57:36.694 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T13:57:36.797 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is off 2024-04-14T13:57:36.804 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-04-14T13:57:36.898 INFO:teuthology.orchestra.console.smithi008:Power off completed 2024-04-14T13:57:40.806 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T13:57:40.916 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is off 2024-04-14T13:57:41.017 INFO:teuthology.orchestra.console.smithi012:Power off completed 2024-04-14T13:57:41.121 INFO:teuthology.run:Summary data: description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} duration: 2091.898464202881 failure_reason: '"2024-04-14T13:39:35.441183+0000 mon.smithi008 (mon.0) 863 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-04-14T13:57:41.121 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-14T13:57:41.218 INFO:teuthology.run:FAIL